Systems and methods for gimbal simulation

ABSTRACT

Systems, devices and methods are provided for training a user to control a gimbal in an environment. The systems and methods provide a simulation environment to control a gimbal in a virtual environment. The virtual environment closely resembles a real control environment. A controller may be used to transmit simulation commands and receive simulated data for visual display.

CROSS-REFERENCE

This application is a continuation application of InternationalApplication No. PCT/CN2015/083788, filed on Jul. 10, 2015, the contentof which is hereby incorporated by reference in its entirety.

BACKGROUND OF THE INVENTION

Flight simulators have been applied in an aviation field for many yearsand are helpful for training pilots or users. In general, the flightsimulators may provide trainees with a virtual scene that is verysimilar to a real scene. Using this virtual scene and auxiliary devices,such as handheld devices, the trainees may be able to virtuallyexperience flight control over an aerial vehicle under simulatedconditions.

However, the current flight simulators mainly provide basic aircraftoperating functions through software applications running on personalcomputers (“PCs”) and fail to provide features that enable a user topractice manipulating a carrier, such as a gimbal, for unmanned aerialvehicles (“UAVs”). The gimbal couples a camera to a UAV to allow the UAVto perform a variety of tasks, including, for example, aerial surveys,aerial imaging, aerial photographs and the like. This lack of gimbalsimulation does not permit the user to train in advance to learn toskillfully control the gimbals.

SUMMARY OF THE INVENTION

A need exists to provide a method of training a user to control flightof an unmanned aerial vehicle (UAV) in a simulated environment. Providedherein are systems and methods to simulate gimbal control.

Thus, in one aspect, a method of simulating gimbal control is provided.The method comprises: receiving, (1) gimbal control data from a remotecontrol system configured to communicate with the gimbal control systemand (2) position data describing a simulated attitude of the vehicle;generating, at a gimbal control system, simulated gimbal response databased on the gimbal control data and the position data describing thesimulated attitude of the vehicle; and transmitting, to the remotecontrol system, the simulated gimbal response data from the gimbalcontrol system on-board the vehicle.

In another aspect, a gimbal simulation system is provided. The gimbalsimulation system comprises: a gimbal on-board a vehicle; a gimbalcontrol system on-board the vehicle configured to (1) receive gimbalcontrol data from a remote control system, (2) receive position datadescribing a simulated attitude of the vehicle generated from a vehiclecontrol system on-board the vehicle; and (3) generate simulated gimbalresponse data based on (i) the gimbal control data and (ii) the positiondata describing the simulated attitude of the vehicle; and acommunication unit configured to transmit the simulated gimbal responsedata to the remote control system.

In another aspect, a method of simulating gimbal control is provided.The method comprises: receiving, at a remote control system remote to avehicle, simulated gimbal response data generated by a gimbal controlsystem on-board the vehicle, wherein the simulated gimbal response datais generated based on (1) gimbal control data from the remote controlsystem configured to communicate with the gimbal control system and (2)position data describing an attitude of the vehicle generated from avehicle control system on-board the vehicle; and displaying, at theremote control system, a simulated gimbal representation based on thesimulated gimbal response data.

In another aspect, a non-transitory computer readable media comprisingprogram instructions for performing a gimbal simulation is provided. Thenon-transitory computer readable media comprises: program instructionsfor processing simulated gimbal response data received at a remotecontrol system remote to the vehicle, said simulated gimbal responsedata generated by a gimbal control system on-board the vehicle, whereinthe simulated gimbal response data is generated based on (1) gimbalcontrol data from the remote control system configured to communicatewith the gimbal control system and (2) position data describing anattitude of the vehicle generated from a vehicle control system on-boardthe vehicle; and program instructions for displaying, at the remotecontrol system, a simulated gimbal representation based on the simulatedgimbal response data.

In another aspect, a method of simulating gimbal control is provided.The method comprises: receiving a gimbal mode signal indicative of aselection from a plurality of gimbal modes; receiving (1) gimbal controldata from a remote control system and (2) position data describing asimulated attitude of the vehicle generated from a vehicle controlsystem on-board the vehicle; and generating, at the gimbal controlsystem, simulated gimbal response data based on (1) the gimbal controldata, (2) the position data describing the simulated attitude of thevehicle, and (3) the gimbal mode signal, wherein the simulated gimbalresponse data causes a different set of axes to be stabilized withrespect to an environment of the vehicle under each of the plurality ofgimbal modes.

In another aspect, a gimbal on-board a vehicle is provided. The gimbalcomprises: a receiver configured to receive a gimbal mode signalindicative of a selection from a plurality of gimbal modes; and a gimbalcontrol system configured to (1) receive gimbal control data from aremote control system, (2) receive position data describing a simulatedattitude of the vehicle generated from a vehicle control system on-boardthe vehicle and (3) simulated gimbal response data based on (1) thegimbal control data, (2) the position data describing the simulatedattitude of the vehicle, and (3) the gimbal mode signal, wherein thesimulated gimbal response data causes a different set of axes to bestabilized with respect to an environment of the vehicle under each ofthe plurality of gimbal modes.

In another aspect, a method of operating a gimbal on-board a vehicle isprovided. The method comprises: receiving a gimbal mode signalindicative of whether the gimbal is to be in an active mode or asimulation mode; receiving gimbal control data from a remote controlsystem; and generating, at the gimbal control system, gimbal responsedata based on the gimbal control data from the remote control system,wherein the gimbal response data is (1) communicated to one or moreactuators configured to adjust an arrangement of the gimbal when thegimbal is in the active mode and is (2) not communicated to one or moreactuators when the gimbal is in the simulation mode.

In another aspect, a gimbal on-board a vehicle is provided. The gimbalcomprises: a receiver, configured to receive a gimbal mode signalindicative of whether the gimbal is to be in an active mode or asimulation mode; a gimbal control system configured to (1) receivegimbal control data from a remote control system, and (2) gimbalresponse data based on the gimbal control data from the remote controlsystem; and one or more actuators configured to (1) adjust anarrangement of the gimbal when the gimbal is in the active mode, or (2)remain dormant and not adjust the arrangement of the gimbal when thegimbal is in the simulation mode.

In another aspect, a method of simulating gimbal control is provided.The method comprises: receiving, (1) gimbal control data from a remotecontrol system configured to communicate with the gimbal control systemand (2) position data describing a simulated attitude of the vehicle;generating, at a gimbal control system, simulated gimbal response databased on the gimbal control data and the position data describing thesimulated attitude of the vehicle; and transmitting, to a displaydevice, the simulated gimbal response data and the simulated attitude ofthe vehicle, wherein the display device generates a visual depictionbased on the simulated gimbal response data and the simulated attitudeof the vehicle.

In another aspect, a gimbal simulation system is provided. The gimbalsimulation system comprises: a gimbal on-board a vehicle; a gimbalcontrol system on-board the vehicle configured to (1) receive gimbalcontrol data from a remote control system, (2) receive position datadescribing a simulated attitude of the vehicle generated from a vehiclecontrol system on-board the vehicle; and (3) generate simulated gimbalresponse data based on (i) the gimbal control data and (ii) the positiondata describing the simulated attitude of the vehicle; and acommunication unit configured to transmit, to a display device, thesimulated gimbal response data and the simulated attitude of thevehicle, wherein the display device generates a visual depiction basedon the simulated gimbal response data and the simulated attitude of thevehicle.

It shall be understood that different aspects of the invention can beappreciated individually, collectively, or in combination with eachother. Various aspects of the invention described herein may be appliedto any of the particular applications set forth below or for any othertypes of movable objects. Any description herein of aerial vehicles,such as unmanned aerial vehicles, may apply to and be used for anymovable object, such as any vehicle. Additionally, the systems, devices,and methods disclosed herein in the context of aerial motion (e.g.,flight) may also be applied in the context of other types of motion,such as movement on the ground or on water, underwater motion, or motionin space.

Other objects and features of the invention will become apparent by areview of the specification, claims, and appended figures.

INCORPORATION BY REFERENCE

All publications, patents, and patent applications mentioned in thisspecification are herein incorporated by reference to the same extent asif each individual publication, patent, or patent application wasspecifically and individually indicated to be incorporated by reference.

BRIEF DESCRIPTION OF THE DRAWINGS

The novel features of the invention are set forth with particularity inthe appended claims. A better understanding of the features andadvantages of the invention will be obtained by reference to thefollowing detailed description that sets forth illustrative embodiments,in which the principles of the invention are utilized, and theaccompanying drawings of which:

FIG. 1 illustrates an example system architecture in accordance with anembodiment of the invention.

FIG. 2 illustrates an example of a simulation method in accordance withan embodiment of the invention.

FIG. 3 illustrates an example of a simulation method under a firstperson view (“FPV”) mode in accordance with an embodiment of theinvention.

FIG. 4 illustrates an example of a simulation method under a followingmode in accordance with an embodiment of the invention.

FIG. 5 illustrates an example of a simulation method under a free gimbalmode in accordance with an embodiment of the invention.

FIG. 6 illustrates an example of a simulation method when environmentfactors are taken into account in accordance with an embodiment of theinvention.

FIG. 7 illustrates an example of a simulation method in accordance withan embodiment of the invention.

FIG. 8 illustrates an example of a simulation method in accordance withan embodiment of the invention.

FIG. 9 is a schematic diagram of a gimbal on-board a vehicle inaccordance with an embodiment of the invention.

FIG. 10 illustrates an example of a method for operating a gimbal inaccordance with an embodiment of the invention.

FIG. 11 is a schematic diagram of a gimbal in accordance with anembodiment of the invention.

FIG. 12 illustrates an example of a simulation method in accordance withan embodiment of the invention.

FIG. 13 is a schematic diagram of a gimbal simulation system inaccordance with an embodiment of the invention.

FIG. 14 schematically illustrates a UAV with a gimbal in accordance withan embodiment of the invention.

FIG. 15 illustrates a movable object including a carrier and a payload,in accordance with an embodiment of the invention.

FIG. 16 is a schematic illustration by way of block diagram of a systemfor controlling a movable object, in accordance with an embodiment ofthe invention.

DETAILED DESCRIPTION OF THE INVENTION

The systems, devices, and methods of the invention provide mechanismsfor training a user to manipulate and control a carrier, which may be agimbal in a virtual or simulated environment. The gimbal may besupported by a vehicle, such as an unmanned aerial vehicle (“UAV”), andmay be used to support a payload. The payload may be an imaging device,which may be used for aerial photography. The gimbal simulation mayoccur while the vehicle is not moving in a real environment (e.g., theUAV is not flying in a real environment). The gimbal simulation mayoccur while the gimbal is or is not moving. The skills and experiencesobtained by the user from manipulating the gimbal in the virtualenvironment may be directly applicable to manipulate a gimbal in a realenvironment. The systems, devices, and methods described herein furtherprovide a simulation platform that employs at least some components thatmay be used for real manipulation of the gimbal. Description of the UAVmay be applied to any other type of unmanned vehicle, or any other typeof movable object.

One or more functions of the gimbal may be controlled at least partiallyby an input from a user. The input from the user may be provided to thegimbal through a remote controller. The remote controller may be used tocontrol flight of the UAV in a real flight environment. The remotecontroller may be used to control movement of a gimbal of the UAV and/orpayload of the UAV in a real flight environment. Providing input tocontrol one or more functions of the gimbal through the remotecontroller may be difficult to novice users. In some cases, a user thatis unfamiliar with providing input to control one or more functions ofthe gimbal through the remote controller may fail to achieve a desiredresult using the remote controller. Failure to achieve good manipulationof the gimbal using the remote controller may result in failure toaccommodate movement of the vehicle in a real flight and make itimpossible to achieve desired aerial photography. Therefore, it may beadvantageous to provide a simulation exercise in which a user can trainand practice controlling a real gimbal in a virtual environment using acontroller system.

In some cases, a virtual or simulated environment may be an outdoor,indoor environment, or mixed outdoor and indoor environment where agimbal may be independently operated, or operated in combination with aUAV to which the gimbal is coupled. The operations of the gimbal in thevirtual environment may be virtual operations of the gimbal. Virtualoperations of the gimbal may or may not include real rotation of one ormore gimbal components about one or more axes. In some instances,virtual operation of the gimbal may include only virtual rotation of theone or more gimbal components about one or more axes. An orientation ofa payload supported by the gimbal may be controlled about one, two,three or more axes. Such control may occur in a real environment or in avirtual environment.

A virtual or simulated environment may be a representation of a realenvironment that exists in real time and space and may be tangible in aphysical world. In some instances, data collected regarding a realenvironment may be used to formulate the virtual or simulatedenvironment. For instance, one or more sensors may collect data about areal environment. Data from the sensors may be used to aid informulating the virtual or simulated environment. An individual mayphysically interact with a real environment. Further, a virtual orsimulated environment may be a mixture of a simulated environment in acomputer software structure and a real environment in a vehicle, such asa UAV. A virtual or simulated environment may be created from one ormore inputs from a user, software developer, or information from adatabase. A virtual or simulated environment may be a representation ofan environment that exists in real space and time or an imaginaryenvironment that does not exist in real space and time. A virtual orsimulated environment may comprise defined boundaries, obstacles, andsurfaces. The virtual or simulated environment may have defined mediumto support manipulation of the gimbal, for example, the medium may beair. The medium may exist and be defined mathematically in the virtualenvironment. In some embodiments, the virtual environment does not existin the physical, tangible world.

A remote controller that is configured to control a real gimbal in avirtual or simulated environment may be the same or similar to acontroller that is used to control a real gimbal in a real environment.In some instances, the simulation remote controller may be the sameactual remote controller that is used to control the gimbal in a realenvironment. In some instances, the simulation remote controller may bean exact duplicate or replica of the remote controller that is used tocontrol the gimbal in a real environment. Any description of a sameremote controller may also apply to a duplicate or replica, or type ofremote controller. In some instances, the simulation remote controllermay have one or more features that are identical to the actual remotecontroller used to control the gimbal in a real environment. Thesimulation remote controller may or may not have one or more featuresthat are different from the actual remote controller used to control thegimbal in a real environment. Providing the same controller for use inboth the simulation and the real environment may result in a morerealistic training experience for a user. A user may develop musclememory associated with movement or manipulation of a physical interfaceon a remote control. Providing an identical remote controller in both asimulation mode and an active mode of a gimbal may provide the advantageof utilizing the muscle memory formed in the simulation mode for use inthe active mode, in which the gimbal may be physically coupled to theUAV and moving an imaging device carried by the gimbal for aerialphotography when the UAV takes off and flies in a space. The musclememory may increase reaction time, precision, and accuracy when thegimbal is in the active mode. Providing the same controller for use inboth the simulation and active modes of the gimbal may familiarize auser with the sensitivity of the controls on the remote control. Forexample, a user may become familiar with the response time of the gimbalto an input from the remote control. In another example, a user maybecome familiar with the magnitude of a response relative to movement ofa physical interface on a remote control. Additionally, a user maymemorize the location of knobs, buttons, joysticks, and/or dials on aremote controller in a simulation mode. As a consequence, when thegimbal is in the active mode, the memorized location of these componentsmay increase reaction time and therefore increase a user's ability tocontrol the gimbal.

In some cases, a UAV, gimbal, and/or payload may be configured toperform autonomous tasks. The autonomous tasks may occur during thegimbal simulation and/or during active operation of the gimbal. Anautonomous task may be initiated by a user, for example, instructing theUAV, gimbal, and/or payload to enter into a gimbal simulation procedure.After an autonomous task is initiated by a user, the UAV, gimbal, and/orpayload may not require additional control or input from a user whilethe autonomous task is performed. An autonomous task may cause the UAVto enter a predetermined sequence. The predetermined sequence mayinclude a series of actions that do not require user input. Forinstance, the UAV may enter a predetermined flight sequence. This mayinclude an automated takeoff sequence, an automated landing sequence, ora predetermined or semi-determined flight path relative to anenvironment (virtual or real) or relative to a reference point (e.g.,ground station, destination point). Another example of an autonomoustask for a UAV may include operation of one or more sensors that maycollect information about an environment of the UAV. In another example,the autonomous task may include communication tasks. The autonomoustasks may relate to operation of the gimbal and/or payload of the UAV.In an example, an autonomous task may be instruction of the gimbal toenter into a simulation mode, and/or instruction of the gimbal to exit asimulation mode. The autonomous task may include transmissions ofposition data (real or virtual) of the UAV to the gimbal in thesimulation mode. The autonomous task may include stabilizing a payloadon the gimbal in accordance with one or more stabilization modes. Theautonomous task may include tracking a (real or virtual) target with thepayload. The autonomous task may include any function of the payload,such as turning the payload on or off, entering a different imagecapture mode (e.g., video vs. still, filtering, color, and lighting),zooming, or any other function.

A gimbal simulation may or may not be coupled with a vehicle motionsimulation. For instance, if a gimbal is supported by a UAV, the gimbalsimulation may be coupled with a flight simulation of the UAV. The usermay have virtual control of the gimbal and/or the UAV during asimulation. In some instances, the user may have virtual control of thegimbal without virtual control of the UAV. For instance, the UAV mayautonomously fly during the simulation, while the user practicescontrolling the gimbal alone. In other instances, the user may controlboth flight of the UAV and operation of the gimbal to simulatesituations when the user will be operating both.

Position of the UAV may affect operation of the gimbal. For instance, anorientation of a payload may be affected by a configuration of thegimbal and an orientation of the UAV. Similarly, a location of thepayload in space may be affected by a configuration of the gimbal and alocation of the UAV in space (e.g., spatial coordinates). Informationabout a UAV position (real or virtual) may be used in aiding the gimbalsimulation.

The position data herein may include simulated aircraft attitude datawhich is descriptive of attitudes of the UAV in a simulation mode, suchas attitude data about a pitch axis, attitude data about a roll axis,and attitude data about a yaw axis, which may be respectivelyrepresented as pitch_aircraft, roll_aircraft, and yaw_aircraft. Theposition data may include simulated aircraft spatial location data whichmay include a location of a UAV in a space in a simulation mode, such asa latitude, longitude, and/or altitude. Simulated aircraft data mayinclude any type of position data. In some cases, the simulated aircraftdata may be derived from real data of the UAV in flight or from one ormore prior flights. The simulated aircraft data may alternatively not bebased on real position data and may be made up as part of the flightsimulation data. The simulated data may or may not depend on an inputfrom a user to control a virtual flight of the aircraft. In some cases,the simulated data may be generated randomly by a computing devicethrough a position data generator running on the computing device. Insome cases, the simulated aircraft data may be semi-static position datathat has been stored by the user in the storage of a computing device,such as a personal computer (PC) or a mobile device, and may be updatedperiodically according to the recent real position data of the UAV. Insome cases, the simulated aircraft data may be stored directly in amemory at any location for use in a simulation. The memory may beon-board the UAV, on-board the gimbal, on-board a payload, on-board aremote controller, on-board a separate device (e.g., server), or part ofa cloud-computing infrastructure. The simulated aircraft data mayinclude any type of aircraft positional data (e.g., attitude data,spatial location data).

During a simulation a user may practice controlling operation of agimbal. The user may directly control virtual operation of the gimbal byproviding one or more input that directly corresponds to a reaction bythe gimbal. For instance, a user may control how one or more gimbalcomponents may rotate. The user may directly be controlling a rotationof a payload about a pitch, yaw, and/or roll axis. A user input maydirectly correspond to a rotation of the payload about one or more axis.A user may control how much the gimbal component rotates to yield acorresponding payload rotation, how fast the rotation is, or anacceleration of the rotation.

A user may indirectly control virtual operation of the gimbal byproviding one or more input that may cause the gimbal to react. Forinstance, the input may cause the gimbal to perform an autonomous taskwithout further intervention or input by the user. In one example, theinput may cause the gimbal to automatically track an object. The payloadmay remain aimed at the object even when the UAV position may change.The gimbal may automatically compensate for the change in the UAVposition.

In some cases, a user may practice instructing a gimbal to perform anautonomous task. The instruction to perform an autonomous task may beprovided to the gimbal in a simulation mode via an identical interfaceas the interface used in the active mode. The interface may be a remotecontroller. The autonomous task may establish communication links with aUAV or with an external computing device or storage to receive simulatedaircraft attitude data. The autonomous task may further perform datafusion directed to combining the position data including the simulatedaircraft attitude data of the UAV and gimbal control data of the gimbalso as to obtain simulated gimbal response data of the gimbal.

A gimbal simulation may or may not make use of actual gimbal data. Aremote controller may communicate with the physical gimbal. Data aboutthe physical gimbal may be used to aid the simulation. The data mayinclude gimbal attitude information, as described further herein. Thegimbal attitude information may include an attitude of a payloadsupported by the gimbal. The gimbal attitude information may include anattitude of one or more gimbal components. The data may includeinformation about signals that are sent to one or more actuators of agimbal. The data may include information about feedback from one or moreactuators of the gimbal. In some instances, virtual gimbal data may beused instead of or in addition to actual gimbal data. For example,virtual gimbal attitude information may be provided based on signalsfrom the remote controller to control one or more actuators of thegimbal. Signals from the physical gimbal may or may not be used in thegeneration of the virtual gimbal attitude information, or any other typeof virtual gimbal data.

The gimbal control data of the gimbal may include real attitude data ofthe gimbal that is collected or sampled when the gimbal is in operation.For example, the real attitude date of the gimbal is collected inreal-time or near real-time when the gimbal is rotating about one ormore of a pitch axis, a roll axis and a yaw axis. This may occur withoutrequiring involvement of the UAV in flight. Therefore, the real attitudedata of the gimbal may be the pitch attitude data, roll attitude dataand yaw attitude data of the gimbal with respect to a UAV to which thegimbal may be coupled. For example, the pitch attitude data, the rollattitude data and the yaw attitude data may be respectively representedas pitch_gimbal_real, roll_gimbal_real, and yaw_gimbal_real.

Gimbal control data may include actual gimbal data and/or virtual gimbaldata. The gimbal control data may be combined with UAV position data.The UAV position data may be UAV virtual position data such as attitudeand/or location. The data may be combined at simulation generationsystem. The simulation generation may include one or more processors.The simulation generation system may be provided on-board the UAV,on-board the gimbal, on-board the payload, on-board the remotecontroller, on-board a remote device, in a cloud-computinginfrastructure, or distributed over any combination.

In some cases, these real attitude data of the gimbal may be transmittedto a remote controller for data fusion with position data including thesimulated attitude data of the UAV.

In some cases, the gimbal control data and position data of the UAV maybe synchronized with each other in a time domain such that the resultingsimulated gimbal response data of the gimbal is much closer to the realenvironment and therefore the user may obtain a better training effect.The user commands may be sent to the gimbal, which may or may notphysically respond, but which may result in gimbal response data beingsent back. By utilizing the physical gimbal, the gimbal response datamay be more realistic in timing and/or substance of response. Thesimulated gimbal response data may be represented pitch_gimbal,roll_gimbal, and yaw_gimbal, denoting relative positions of the gimbalwith respect to a UAV-body coordinate frame, or denoting absolutepositions of the gimbal with respect to an earth-surface inertialreference frame.

The gimbal control simulation may use a display device to depict avirtual simulated environment of the gimbal. An application may run on adisplay device. The display device may be part of a remote controllerthat may control operation of the gimbal. The display device may be aseparate device from the remote controller. The display device may ormay not communicate with the remote controller. The display device mayoptionally be detachably mounted on the remote controller, or have ashared substrate with the remote controller. The display device may bephysically separated from the remote controller and may or may notwirelessly communicate with the remote controller. The display deviceand remote controller may be part of a remote control system.

The application may cause the display to show a three-dimensionalvirtual environment. The virtual environment may be shown from a thirdparty view that may also show the UAV and/or gimbal. The virtualenvironment may be shown from a perspective of an imaging devicesupported by the gimbal. The display device may show movement of thegimbal within the environment. This may be shown from a perspectiveoutside the gimbal which may show how the various gimbal components maymove relative to one another and/or the environment. Further, theapplication may also show a video image dynamically varied with theresulting simulated attitude data of the gimbal. In some instances, thedisplay device may simultaneously show multiple views. For instance, themultiple views may include a view from a perspective outside the UAVand/or gimbal, and a view from the imaging device's perspective. In thismanner, the manipulation performed by the user on the gimbal may bevisualized and the user may be able to intuitively adjust the movementof the gimbal such that desired images or pictures may be captured inthis simulation environment.

As previously described, a vehicle control system on-board a vehicle maybe used to control movements or attitudes of the vehicle. When thevehicle is a UAV, then the vehicle control system may be a flightcontrol system that may be used to control flight of the UAV within thevirtual or simulated environment. The vehicle control system may be partof a simulation generation system. The vehicle control system mayinclude one or more processors. The vehicle control system may beon-board the UAV, on-board the display device, on-board the payloadon-board the remote controller, on-board a separate device, part of acloud computing infrastructure, or distributed any of these. The vehiclecontrol system may use data from virtual sensors or real sensors togenerate a simulated flight and simulated attitude data of the UAV. Insome instances, a UAV may operate within a flight mode or a simulatedmode. When in the flight mode, the UAV flight control system may sendsignals to propulsion units of the UAV to effect flight of the UAV. Whenin a simulated mode, the UAV flight control system may send signals to aphysical model without sending signals to the propulsion units. Thephysical model may provide virtual feedback, which may help define thesimulated flight of the UAV. The same flight control system may be usedduring real flight of the UAV and simulated flight of the UAV.

Further, as previously described, a remote control system may be used tocontrol movements of the gimbal within the virtual or simulatedenvironment. The remote control system may include a remote controllerconfigured to accept a user input to effect control of the gimbal. Theremote control system may be configured to communicate with a gimbalcontrol system on-board the UAV and/or gimbal. The remote control systemmay also be configured to receive the simulated attitude data of the UAVand transmit the simulated attitude data of the gimbal to a displaydevice for visual display. The simulation generation system may or maynot be the remote control system. The simulation generation system maybe on one or more separate devices from the remote control system.

Provided herein are systems, methods, and devices configured to providea realistic gimbal simulation. A realistic gimbal simulation may be asimulation for gimbal control that comprises components used in a realoperation of a gimbal. Possible components of a realistic gimbalsimulation system are shown in FIG. 1. A realistic gimbal simulationsystem may comprise a remote controller 101, a display device 102, aconnector 103 between the remote controller and the display device, avehicle 104, such as a UAV, a gimbal 105 supported by the vehicle, and apayload, such as an image capture device 106 supported by the gimbal.

A remote control system may communicate with a gimbal control system.The remote control system may include a remote controller and/or adisplay device. As illustrated, the remote controller and display devicemay be separate devices. The remote controller and display device may beconnected to one another via a connector. The connector may be aflexible cable. The remote controller and display device may be directlyconnected to one another. The remote controller and display device maybe removably directly connected to one another or permanently connectedto one another. The remote controller and display device may beindirectly connected to one another via one or more intermediarydevices. The remote controller and display device may be removablyindirectly connected to one another or permanently connected to oneanother. The remote controller and display device may be physicallyseparated from one another. The remote controller and display device maybe in communication with one another. The remote controller and displaydevice may be in communication via a hard-wired connection or viawireless connections. In some instances, direct wireless communicationsmay be provided. Alternatively, indirect wireless communications may beprovided, such as communications with one or more intermediary devices(e.g., base stations, satellites, towers), or over a network. The remotecontroller and display device may be the same device. A remotecontroller may both accept inputs to affect operation of the UAV and/orgimbal and display information pertaining to the simulation. The remotecontrol system may encompass a single device or multiple devices.

The remote controller may be used to control operation of a gimbal in anactive mode or simulation mode. The remote controller may be the sameremote controller that is used to control a UAV in a real flightoperation or in a simulation mode. In some cases the remote controllermay be a similar or identical copy of a remote controller that is usedto control a gimbal and/or a UAV in a real flight operation. The remotecontroller may have any combination of physical user interfacemechanisms. A physical user interface mechanism may be a component onthe remote controller that a user touches or manipulates to control atleast one function of the gimbal and/or UAV. In an example, a physicaluser interface mechanism may be a button, a joystick, a lever, a rollerball, a touch screen, a switch, a dial, or a knob. The physicalinterface may include one or more inertial sensors that may measure anattitude of the remote controller. In some cases, the physical userinterface may comprise two or more joysticks. The joysticks may movevertically and/or horizontally. The joysticks may move both verticallyand horizontally The joysticks may be used to control pitch, roll, andyaw and therefore the physical user interface mechanisms may beconfigured such that a user can control movement of the gimbal about aroll, yaw, and/or pitch axis as depicted in FIG. 1. The physical userinterface mechanisms may be further configured to provide a user controlover operations of the gimbal and/or UAV in the simulated environment.In some embodiments, one or more controls may be used for controllingthe gimbal and the UAV. The controls may be physical or virtualcontrols. For example, the physical controls may be the same set ofjoysticks that may be used to control the gimbal and the UAV and thevirtual controls may be virtual direction keys for controlling thegimbal and the UAV. In other embodiments, different sets of joysticksmay be used to control the gimbal and the UAV. Similarly, the samephysical user interface may be used to control the gimbal and the UAV.Alternatively, different physical user interfaces may be used to controlthe gimbal the UAV. The same remote controller may be used to controlthe gimbal and the UAV. In some embodiments, a gimbal remote controllerand a UAV remote controller may be provided as separate devices. Thegimbal remote controller and the UAV remote controller may or may not bephysically connected to one another. The gimbal remote controller andthe UAV remote controller may or may not be in communication with oneanother. A gimbal remote controller and a UAV remote controller may beconfigured to be operated simultaneously by the same user.Alternatively, the gimbal remote controller and the UAV remotecontroller may be configured to be operated by different userssimultaneously. In some instances, for gimbal simulation, only controlsfor the gimbal are provided. Alternatively, controls for both the gimbaland the UAV may be provided.

In some cases, the physical user interface may provide mechanisms tocontrol non-flight actions of the UAV. A non-flight action may bemovement of a sensor or payload on-board the UAV. The non-flight actionmay also include actuation of a gimbal of the UAV that may be configuredto carry a payload. Another example of a non-flight action can becollection and/or reporting of the simulated attitude data previouslycollected by a sensor on-board the UAV. Additionally, the physical userinterface may provide mechanisms to initiate an autonomous action ortask by the UAV. In an example, an autonomous task or action may betransmissions of the position data including the simulated attitude databy the UAV to a remote control system in the remote controller. In someembodiments, controlling the operations of the gimbal 105 may includecontrolling rotational acceleration of one or more components of thegimbal, rotational speed of one or more components of the gimbal,attitudes of one or more components of the gimbal, and/or power to thegimbal or more components of the gimbal.

In accordance with some embodiments, the remote controller may connectto a display device through a wired or wireless connection. Any otherconfiguration or relation of the remote controller and display devicemay be provided, as previously described.

The display device may be a device that comprises a computing componentand a visual display. The computing component may comprise one or moreprocessors, and one or more memory storage devices. The processors maybe configured to execute instructions in accordance with non-transitorycomputer readable medium. The memory may comprise non-transitorycomputer readable media comprising code, logic, or instructions forperforming one or more steps described herein. The display device maycomprise a non-transitory computer readable media comprising programinstructions for performing a gimbal control simulation. The displaydevice may be a mobile device, such as a smart phone. In some cases, thedisplay device may be a desktop computer, laptop computer, tablet, orvirtual reality headset. Alternatively, the display device may be acombination of a computing component and a visual display where a visualdisplay may be a touchscreen, projector, LCD screen, plasma screen, LEDor OLED screen, a television, or a monitor.

The display device may provide a visual and/or textual representation ofsimulated attitude data of the gimbal during a gimbal controlsimulation. In some cases, the display device may additionally provideaudio feedback during a gimbal control simulation. The display devicemay be configured to receive user input through a user interactivecomponent, such as a touchscreen, switch, button, key, knob, mouse,pointer, trackball, joystick, touchpad, inertial sensors (e.g.,accelerometers, gyroscopes, magnetometers) microphone, visual sensor, orinfrared sensor. The user interactive component may receive touchinputs, positional inputs, audio inputs, or visual inputs.

The remote controller may be in communication with the display device.Communication between the remote controller and the display device maybe provided through a wired or wireless connection. A wirelessconnection may be provided between the remote controller and the displaydevice through an RF connection, IR connection, Wi-Fi network, awireless local area network (WLAN), a cellular network, or any otheravailable wireless network. Additionally or alternatively, a wiredconnection may be provided between the remote controller and the displaydevice through a permanent wire connection, coaxial cable connection,Firewire connection, MIDI connection, eSTATA connection, an Ethernetconnection, or any other available wired connection that permits datatransmission. In some cases, the wired connection may be a connectionthrough a USB cable 103 via USB ports. In some cases, the display deviceis integrated in the remote controller and become a part of the remotecontroller. Therefore, it would be easy for the user to observe thesimulation of the gimbal control by merely holding the remotecontroller.

In some embodiments, the remote controller and/or the display device maybe in communication through a wired or wireless connection with avehicle control system or flight controller. The vehicle control systemmay be on-board or off-board the UAV. In some embodiments, the vehiclecontrol system may be on-board the remote controller or the displaydevice. The vehicle control system may be configured to generateposition data describing a simulated attitude of the UAV in response toan input from the remote controller and/or the display device. Theposition data may or may not include a simulated spatial location of theUAV. The vehicle control system may receive input from a user throughthe remote controller and/or the display device. The vehicle controlsystem may communicate the input to a system of one or more componentsthat may generate real or virtual sensor data and communicate this databack to the vehicle control system. Based on the real or virtual sensordata, the vehicle control system may generate position data, such assimulated attitude data of the UAV, and transmit this simulated data tothe simulation generation system.

The remote control system may be remote to the UAV and/or the gimbal.The remote control system may be remote to the gimbal control system.The location of the UAV and/or the gimbal may be independent of thelocation of the remote control system. In some instances, there may be alimited range between the UAV and/or gimbal and the remote controlsystem. Alternatively, there may be no limit to the range between theUAV and/or gimbal and the remote control system.

The remote controller and/or the display device may be in communicationthrough a wired or wireless connection with a gimbal control system. Thegimbal control system may be on-board or off-board the gimbal and/orUAV. In some embodiments, the gimbal control system may be on-board theremote controller or the display device. The gimbal control system maybe configured to generate position data describing a simulated attitudeof the payload or one or more gimbal components in response to an inputfrom the remote controller and/or the display device. The attitudeinformation may be with respect to a virtual environment or the UAV. Thegimbal control system may receive input from a user through the remotecontroller and/or the display device. The gimbal control system maycommunicate the input to a system of one or more components that maygenerate real or virtual sensor data and communicate this data back tothe gimbal control system. Based on the real or virtual sensor data, thegimbal control system may generate position data, such as simulatedattitude data of the gimbal, and transmit this simulated data to thesimulation generation system. The gimbal may or may not be moving inresponse to the user inputs.

The simulation generation system may include the remote controller. Thesimulation generation system may be at any other device, as describedelsewhere herein, or distributed over multiple devices. At thesimulation generation system, data may be combined to generate simulatedgimbal response data. The simulated gimbal response data may include thesimulated attitude data of the gimbal. The simulated attitude data ofthe gimbal may be respect to the environment. The simulated UAV positioninformation may be combined with the simulated gimbal positioninformation (e.g., with respect to the UAV) to generate the simulatedgimbal response data. The following will describe the method ofobtaining position data including simulated attitude data of the UAV andgimbal control data including the real attitude data of the gimbal.

The UAV may be operated in a first or second operation mode. In thefirst operation mode, the UAV may fly in a real environment by receivinginstructions or input from a remote controller 101. Hence, the firstoperation mode may be a flight mode or active mode.

The second operation mode may be a simulation mode. In a secondoperation mode, the UAV may remain physically dormant and may not beself-propelled within the real environment. One or more propulsion units107 of the UAV, may not operate while the UAV is in the simulation mode,which may be consistent with a simulation mode of the gimbal asdiscussed later. In the simulation mode, one or more components on-boardthe UAV may contribute to a gimbal control simulation.

When the UAV is operating in a flight mode, the remote controller mayprovide an input to the vehicle control system. The input provided bythe remote controller may be flight control data. Flight control datamay be an instruction that changes a flight path or causes a flightevent to start or stop. In an example, flight control data may be aninstruction to start a propulsion system, stop a propulsion system,increase power to a propulsion system, decrease power to a propulsionsystem, change the heading of a UAV, change the elevation of the UAV,turn on a sensor on a UAV, turn off a sensor on a UAV, report sensordata from a sensor on-board the UAV, or initiate an autopilot functionon the UAV. The vehicle control system may receive and process theflight control data using one or more processors. The processors may beconfigured to, individually or collectively, transform the flightcontrol data into an instruction to alter, initiate, or cease a flightaction. The processors may transform the flight control data identicallyin both flight and simulation modes of operation.

When the UAV 104 is in the flight mode, the flight control data may becommunicated to one or more propulsion units of the UAV. A vehiclecontrol system (e.g., on-board the UAV) may be configured to generateone or more flight signals to be communicated to the one or morepropulsion units when the UAV is in the flight mode. When the UAV is inthe flight mode, the one or more propulsion units may be configured toactuate and permit flight of the UAV in response to the flight signals.The one or more propulsion units may further be configured to remaindormant and not permit flight of the UAV when the UAV is in thesimulation mode in which the one or more propulsion units may notreceive a flight signal.

Optionally, in a flight mode, the remote controller may be configured tocontrol actuation of a carrier, such as the gimbal that holds a payloadof the UAV. The gimbal may be permanently affixed to the UAV or may beremovably attached to the UAV. The gimbal may include one or more gimbalcomponents that may be movable relative to one another. The gimbalcomponents may rotate about one or more axis relative to one another.The gimbal may include one or more actuators that effect rotation of theone or more gimbal components relative to one another. The actuators maybe motors. The actuators may permit rotate in a clockwise and/orcounter-clockwise direction. The actuators may or may not providefeedback signals as to the position or movement of the actuators. I someinstances, one or more gimbal component may support or bear the weightof additional gimbal components. In some instances, gimbal componentsmay permit rotation of a payload about a pitch, yaw, and/or roll axis. Agimbal component may be permit rotation about a pitch axis, anothergimbal component may permit rotation about a yaw axis, and anothergimbal component may permit rotation about a roll axis.

The gimbal may support a payload. The payload may be permanently affixedto the gimbal or may be removably attached to a gimbal. The payload maybe supported by a gimbal component. The payload may be directedconnected to the gimbal component. The payload may remain at a fixedposition relative to the gimbal component Alternatively, the payload mayrotate relative to the gimbal component.

A payload may be an external sensor, for example a camera unit includingthe image capture device. The image capture device may be movableindependent of the motion of the UAV. The image capture device may bemovable relative to the UAV with aid of the gimbal. Similar to thepropulsion units, when the UAV is in the flight mode, a carrier,payload, sensor, and/or other component of the UAV may receive, from oneor more control systems on-board the UAV a variety of control signalswhich may cause corresponding operations directed to the carrier,payload, sensor, and/or other component. When the UAV is in thesimulation mode, a user may practice controlling direction of a cameraon-board the UAV without practically activating the UAV to take off.When the gimbal is in a simulation mode, the user may practicecontrolling direction of the camera without practically activating thegimbal to rotate. Alternatively, in the simulation mode, the gimbal maystill rotate.

In some embodiments, the UAV may participate in the gimbal controlsimulation. When the user desires to simulate gimbal control, the UAVand/or gimbal may be turned on by a user instruction from the remotecontroller. The gimbal control system or the vehicle control system maytransmit the position data, which may include simulated attitude data ofthe UAV and/or simulated attitude data of the gimbal to a simulationgeneration system. The simulation generation system may include theremote control system in the remote controller. At the simulationgeneration system, data fusion could be performed for generating thesimulated gimbal response data. Alternatively or additionally, thesimulation generation system may include a gimbal control systemon-board the gimbal and/or the UAV and then the resulting simulatedgimbal response data may be transmitted to the remote control system viaa wired or wireless connection as discussed before. The remote controlsystem may include the remote controller and/or the display device 102.The simulated gimbal response may be displayed on the display device.

In some embodiments, the UAV may comprise a receiver configured toreceive a mode signal that indicates that the UAV is in a first orsecond mode. The mode signal may be provided by the remote controller,the display device, or a separate device in communication with thereceiver. In some cases, the signal may be provided through a hardwarecomponent on the UAV. The hardware component may be manipulated by auser to provide the signal to the UAV. For example, the hardwarecomponent may be a switch, button, or knob that may be physicallydisplaced between a first and second position to provide a signalindicating a first or second mode. In another example, a flight mode maybe a default mode for the UAV and the UAV may operate in the flight modeunless a mode signal indicates a change to the simulation mode in whichthe UAV operates to facilitate the gimbal control simulation. Asdescribed previously, the UAV may transmit its simulated position datato a simulation generation system (e.g., remote control system, gimbalcontrol system). Additionally or alternatively, the simulationgeneration system may be on-board the UAV, and the UAV may perform datafusion on its own based on the gimbal control data received from theremote controller.

The gimbal may or may not have a separate receiver configured to receivea mode signal that indicates whether the gimbal is in the first mode orthe second mode. The mode signal may be provided by the remotecontroller, the display device, or a separate device in communicationwith the gimbal receiver. In some cases, the signal may be providedthrough a hardware component on the gimbal. The hardware component maybe manipulated by a user to provide the signal to the gimbal. Forexample, the hardware component may be a switch, button, or knob thatmay be physically displaced between a first and second position toprovide a signal indicating a first or second mode. In another example,an active mode may be a default mode for the gimbal and the gimbal mayoperate in the active mode unless a mode signal indicates a change tothe simulation mode. In some embodiments, a UAV may automaticallycommunicate a mode selection to the gimbal. The gimbal mode may beautomatically updated to match the UAV mode. For instance, if the UAV isin a flight mode, the gimbal may automatically be in active mode. If theUAV is in a simulation mode, the gimbal may also be in a simulationmode.

In some embodiments, none of the components on-board the UAV may be usedin the gimbal control simulation. For example, in a simulation mode, avirtual UAV may be flown in a virtual or simulated environment andtherefore the position data including the simulated attitude data of theUAV may be obtained or generated in such an environment. The virtual UAVand the virtual environment may exist mathematically in a virtual orsimulated space, such as one established in a computer environment. Thevirtual UAV may have the same functionality in the virtual environmentas the real UAV in the real environment. In other words, a real UAV isnot required to implement the simulation for gimbal control. The gimbalcontrol simulation may be implemented in many different and flexibleways without the real UAV as long as the position data that includes thesimulated attitude data of the UAV is available. The simulated attitudedata of the UAV may or may not come from a real UAV. The simulatedattitude data for the UAV may or may not be generated by a vehiclecontrol system of the UAV. The simulated attitude data for the UAV maybe generated with aid of one or more processors based purely on avirtual UAV.

In some embodiments, the gimbal 105 may be operated in a first operationmode or a second operation mode. In the first operation mode, the gimbalmay be supported by the UAV 104 and may support an image capture device106. Further, the gimbal may have been turned on and ready for carryingthe image capture device for aerial photography rather than performingsimulation operations. Optionally, the image capture device may bepowered on when the gimbal is in the first operation mode. The imagecapture device may be configured to communicate captured images in astreaming manner in the first operation mode. Therefore, the firstoperation mode of the gimbal herein may be an active mode.

In the second operation mode, the gimbal 105 may or may not be supportedby the UAV 104. Further, the gimbal may or may not support the imagecapture device. Although the gimbal may adjust its arrangement and insome embodiments, if the image capture device is mounted, drive theimage capture device to rotate and move by one or more actuatorsarranged on the gimbal, the movements of the gimbal herein are merelyintended to yield the gimbal control data including the real attitudedata of the gimbal rather than engendering the image capture device toperform practical aerial photography. In other words, the secondoperation mode of the gimbal is a simulation mode in which the gimbalcontrol data including the real gimbal attitude data of the gimbal maybe collected for generating the simulated gimbal response data. This mayprovide realistic feedback for the user, such as how much the gimbalwill end up physically moving in response to a user input. The simulatedUAV attitude data may be used in combination in generating the simulatedgimbal response data. In some embodiments, the gimbal does not adjustits arrangement, and the image capture device is not rotated.Information from real sensors on-board the gimbal, may or may not beused as virtual gimbal attitude data. In some instances, communicationmay occur with a gimbal control system. The gimbal control system maysend instructions to gimbal actuators to move in an active mode. In asimulation mode, the gimbal control system may not actually send thesignal to cause the actuator to move, but may generate the signals andgenerate simulated gimbal attitude data.

In the second operation mode, the image capture device may or may not bepowered on. In some instances, the image capture device is powered off.In the second operation mode, the image capture device may or may not berecording or streaming image data captured by the image capture device.In some instances, the image capture device is not recording orstreaming the image data in the simulation mode. Instead, virtual imagedata may be generated and displayed at a display device. The virtualimage data may be reflective of a virtual position of the image capturedevice within the virtual environment. The virtual position of the imagecapture device with respect to the virtual environment may be reflectiveof the simulated gimbal response data.

In some embodiments, the gimbal may include a receiver through which auser may initiate a mode switch between an active mode and a simulationmode of the gimbal. In an example, a user may choose to use the gimbalin a simulation mode. To this end, the user may provide a mode switchingsignal or a mode selecting signal to the receiver to indicate that thegimbal should operate in a simulation mode. The user may provide themode switching signal via a physical interface on the gimbal (e.g. aswitch, a button, a lever, or a knob). In some cases, the user mayprovide the mode switching signal through the remote controller via aphysical interface mechanism on the remote controller. In some cases, analternate or remote device, which may be different from the remotecontroller used for gimbal or UAV control, may be used to send a modeswitching signal to the gimbal.

Additionally or alternatively, the display device may be used to send amode switching signal to the gimbal. For example, when the displaydevice is turned on, it may automatically connect to a communicationunit arranged on the gimbal. The gimbal may automatically enter into thesimulation mode by default whenever the display device is incommunication with the gimbal. In some cases, the gimbal may notautomatically enter into the simulation mode and the user maycommunicate a mode switching signal to the receiver arranged on thegimbal through the display device by using, for example, a touch screenon the display device.

When the receiver arranged on the gimbal receives a mode switchingsignal to switch the mode from the simulation mode to the active mode,the gimbal may immediately cease the simulation mode and turn into theactive mode, in which the gimbal gets prepared for operation with theUAV while the UAV is in flight to perform the aerial photography via theimage capture device.

Optionally, in a simulation mode, the remote controller may beconfigured to control actuation of a carrier (e.g., the gimbal) thatholds a payload of the UAV (e.g., the image capture device). A payloadmay be an external sensor, for example a camera unit. The payload may bemovable independent of the motion of the UAV. Optionally, in thesimulation mode, the remote controller may be configured to controlactuation of the carrier (e.g., the gimbal), which may or may not bephysically on-board the UAV. For example, a user may practicecontrolling directions of a camera on-board a UAV in a gimbal controlsimulation using a gimbal control system that may be on-board the UAV,the gimbal, the remote controller, the display device, or other device.In another example, the user may practice controlling directions of acamera supported by a gimbal that may not be on a UAV or other vehicle.The user may thus focus on practicing the gimbal control alone withoutworrying about other UAV functions, or while allowing the UAV functionsto be simulated.

A gimbal simulation system may be provided. The gimbal simulation systemmay comprise the gimbal on-board the vehicle. The gimbal simulationsystem may further comprise a gimbal control system on-board the gimbal,the vehicle, the payload, or any other device or component describedherein. The gimbal control system may be configured to receive gimbalcontrol data from a remote control system and receive position datadescribing a simulated attitude of the vehicle. The simulated positionof the vehicle may be generated from a vehicle control system on-boardthe vehicle or may be virtually generated without requiring the physicalvehicle. The simulated position may be at least partially based on asimulated orientation of the vehicle. Further, the position datadescribing an attitude of the vehicle from the vehicle control systemon-board the vehicle may be at least partially based on simulatedweather data.

In some embodiments, the position data describing an attitude of thevehicle is at least partially based on the position data describing thesimulated attitude of the vehicle which includes at least one of (1)rotation of the vehicle about a pitch axis, (2) rotation of the vehicleabout a yaw axis, or (3) rotation of the vehicle about a roll axis. Theposition data may include a simulated spatial location of the vehicle.The simulated spatial location of the vehicle may include at least oneof (1) a latitude, (2) a longitude, or (3) an altitude of the vehicle.The simulated spatial location of the vehicle may be generated based onsimulated output to the propulsion units. The simulated spatial locationmay at be further based on simulated weather data.

The gimbal control system may generate simulated gimbal response databased on (i) the gimbal control data and (ii) the position datadescribing the simulated attitude of the vehicle. The gimbal controldata may or may not incorporate data from the physical gimbal or thegimbal control system used to control the physical gimbal. The gimbalsimulation system may additionally comprise a communication unitconfigured to transmit the simulated gimbal response data to the remotecontrol system. The simulated gimbal response data herein may begenerated by one or more processors. The simulated gimbal response datamay be generated at a simulation generation system.

In some embodiments, the simulated gimbal response data is determinedbased on a gimbal mode signal. The gimbal mode signal may be generatedat the remote control system remote to the vehicle. Further, the gimbalmode signal is generated in response to a user input indicating aselection of a gimbal mode from the plurality of gimbal modes. Thesimulated gimbal response data may be obtained by a display devicecomprising a visual display showing the simulated gimbal stateinformation of the gimbal. The simulated gimbal state informationincludes simulated visual data captured by a camera supported by thegimbal. The remote control system may include a remote controller usedto operate the vehicle and/or the gimbal in a real flight operation. Theremote controller may include one or more joystick controls useful forcontrolling directional heading of the gimbal.

In some embodiments, the gimbal simulation system may further comprise adisplay device that receives the simulated gimbal response data anddisplays a visual illustration of the gimbal in an orientation describedby the gimbal response data. The display device may be part of theremote control system. The display device may be integrated with theremote controller, may be connected to the remote controller, or may becommunicating with the remote controller. The display device may be ahandheld device in communication with the gimbal control system. Thedisplay device may directly communicate with the gimbal control systemor may communicate with the gimbal control system through the remotecontroller. The remote controller may directly communicate with thegimbal control system or may communicate with the gimbal control systemthrough the display device. The remote controller and/or display devicemay directly communicate with the gimbal control system or maycommunicate with the gimbal control system through the UAV or payload.

According to the embodiments of the invention, the gimbal controlsimulation may enable the user to be more familiar with utilization ofthe gimbal 105 to control the direction in which the image capturedevice is currently directed for aerial photography.

FIG. 2 is a flow chart schematically illustrating a method of simulatinggimbal control in accordance with an embodiment of the invention. Any ofthe steps are optional and/or may be performed in different orders. Anyof the steps may be exchanged for other steps.

The method of simulating gimbal control may include obtaining gimbalcontrol data and vehicle position data 201. Simulated gimbal responsedata may be generated based on the gimbal control data and the positiondata 202. The simulated gimbal response may be transmitted to a remotecontrol system 203.

The gimbal control data and the vehicle position data may be obtained201. The gimbal control data may be sent from a remote control system.The remote control system may be configured to communicate with a gimbalcontrol system. The gimbal control data may be generated based on aninput of a user at a remote controller of the remote control system. Theinput of the user may be a direct control of the gimbal or may be anindirect control of the gimbal that may initiate an automated action bythe gimbal. The gimbal control data may or may not include measurementsfrom the gimbal in response to the user input. The gimbal control datamay or may not include a simulated gimbal attitude with respect to avehicle in response to the user input. The vehicle position data maydescribe a simulated attitude of a vehicle. The simulated attitude ofthe vehicle may be generated by a vehicle control system of a vehicle,or with aid of any physical component of the vehicle. Alternatively, thesimulated attitude of the vehicle may be entirely virtually generatedwithout requiring a physical vehicle or vehicle control system of thevehicle. The vehicle position data may or may not include a simulatedspatial location of the vehicle. The simulated spatial location of thevehicle may be generated by the vehicle control system of the vehicle,or with aid of any physical component of the vehicle. Alternatively, thesimulated spatial location may be entirely virtually generated withoutrequiring a physical vehicle or vehicle control system of the vehicle.

Simulated gimbal response data is generated based on the gimbal controldata and the vehicle position data 202. The vehicle position data maydescribe the simulated attitude of the vehicle. The vehicle positiondata may or may not describe the simulated spatial location of thevehicle. The simulated gimbal response data may be generated at a gimbalcontrol system. The simulated gimbal response data may be generated at asimulation generation system, which may or may not include the gimbalcontrol system. The simulated gimbal response data may includeinformation about position of the gimbal. The information about theposition of the gimbal may be relative to a virtual environment. Forinstance, the virtual position of the gimbal relative to the virtualenvironment may be generated based on a virtual position of the gimbalrelative to the UAV and a virtual position of the UAV relative to thevirtual environment. The virtual positions may include attitudeinformation and/or spatial position information. In other embodiments,the virtual position of the gimbal may be relative to the virtualenvironment or the UAV, and may be generated based on user input on howto control the gimbal and the virtual position of the UAV relative tothe virtual environment. The virtual position of the gimbal may or maynot take a previously performed calculation of the gimbal control systeminto account.

The simulated gimbal response data may be obtained at the remote controlsystem 203. The simulated gimbal response data may be transmitted fromthe gimbal control system. The simulated gimbal response data may betransmitted from a simulation generation system, which may or may notinclude the gimbal control system. The simulation generation system mayalready be at the remote control system, which may not require separatetransmission. In some embodiments, the simulated gimbal response dataincludes simulated gimbal state data which represents an attitude of thegimbal relative to the vehicle or the virtual environment. The simulatedgimbal response data may be obtained at a remote controller and/ordisplay device of the remote control system. The same device that may beused to control the gimbal may also comprise a visual display.

The remote control system may control the actuations, rotations andmovements of the gimbal such that the gimbal control data including realattitude data of the gimbal may be obtained. The remote control systemmay include one or more joystick controls useful for controllingdirectional heading of the gimbal. Further, in some embodiments, theremote control system may comprise a computer system which maycommunicate with the gimbal using a wired connection or a wirelessconnection. The gimbal control data collected by a physical model mayrun on the computer system that may comprise one or more processors andone or more memory storage units. The physical model may be run inaccordance with non-transitory computer readable media. Thenon-transitory computer readable media may comprise code, logic, orinstructions for performing one or more steps described herein. Theprocessors may, individually or collectively, execute steps inaccordance with the non-transitory computer readable media. The remotecontrol system may include a visual display. The visual display maycommunicate with the computer system.

In some embodiments, a visual display may show simulated gimbal stateinformation. The simulated gimbal state information may includeinformation about a position of a gimbal. The state of the gimbal mayinclude whether the gimbal is on or off, a stabilization mode of thegimbal, positioning of one or more components of the gimbal, positioningof a payload supported by the gimbal, movement characteristics (e.g.,speed, acceleration) of one or more components of the gimbal, or spatialposition of the gimbal. The visual display may or may not showinformation about the vehicle. The visual display may show informationabout a state of the vehicle. The state of the vehicle may includewhether the vehicle is on or off, positioning of the vehicle, movementcharacteristics (e.g., speed, acceleration) of the vehicle, powerconsumption of the vehicle, data collected by the sensors of thevehicle, information about operational parameters of the vehicle,information about communications of the vehicle, or navigationalinformation of the vehicle. The visual display may or may not showinformation about the vehicle. The visual display may include simulatedvisual data captured by a camera supported by the gimbal.

The simulated visual data may be generated based on simulated positioninformation of the gimbal. This may include simulated positioninformation relative to the virtual environment. The simulated visualdata may also depend on the simulated environment. For instance,simulated geographic features or objects may be displayed. The virtualposition of the gimbal (e.g., orientation, spatial coordinates) maydetermine which angle the camera views the virtual environment, andthus, how the simulated visual data is displayed.

The simulated visual data may also be generated based on a stabilizationmode of the gimbal. A gimbal stabilization mode may optionally beselected from a plurality of available stabilization modes. The selectedgimbal stabilization mode may affect how the gimbal is virtuallypositioned (e.g., by determining which rotational axes the payload isstabilized/independent of vehicle movement).

In some embodiments, simulated gimbal response data may be determinedaccording to a gimbal mode (i.e., gimbal stabilization mode), whereinthe gimbal mode includes one of a first person view mode, a followingmode, or a free gimbal mode. When the gimbal is operated in the firstperson view mode, the simulated gimbal response data stabilizes a pitchaxis of the gimbal with respect to the environment without stabilizing ayaw axis and a roll axis. When the gimbal is operated in the followingmode, the simulated gimbal response data stabilizes a pitch axis and aroll axis of the gimbal with respect to the environment withoutstabilizing a yaw axis. When the gimbal is operated in the free gimbalmode, the simulated gimbal response data stabilizes a pitch axis, a yawaxis, and a roll axis of the gimbal with respect to the environment.

It should be noted that the gimbal modes herein, including the firstperson view mode, the following mode, and the free gimbal mode, are onlyillustrative of multiple modes that the gimbal may have and the gimbalmodes should not be limited to these three specific forms. A personskilled in the art may contemplate, based on the teaching of theinvention, that the gimbal may operate in other modes and therefore theresulting simulated gimbal response data may also be used to stabilizeone or more of the pitch axis, the yaw axis and the roll axis asnecessary. Any of the alternative modes may stabilize one of the pitch,yaw, or roll axis, stabilize any two of the pitch, yaw, or roll axes, orstabilize all three of the pitch, yaw, and roll axes. The degree ofstabilization, or response time for stabilization for any of the axesmay also vary.

The simulated gimbal response data may be determined based on a gimbalmode signal. The gimbal mode signal may be generated at a remote controlsystem. For instance, the gimbal mode signal may be generated at aremote controller that may also accept user input to control operation(virtual or real) of the gimbal. The gimbal mode signal indicates a thegimbal mode to be followed by the gimbal, such as the first person viewmode, the following mode and the free gimbal mode as discussed before.Further, in some embodiments, the gimbal mode signal is generated inresponse to a user input indicating a selection of a gimbal mode fromthe plurality of gimbal modes. The user may be able to view multipleoptions for various modes, and may be able to select an option from themultiple options. In some instances, visual representations of thevarious gimbal modes may be displayed on the screen. The user may beable to touch the screen to select a desired mode, or use some sort oftracking device (e.g., mouse, joystick, trackball, touchpad, button) toselect the desired mode.

The user may be able to select a gimbal operation mode while the gimbaland/or vehicle are in an active mode. Thus, when the vehicle is inmotion (e.g., UAV in flight), the user may be able to select a gimbalmode that may affect the stabilization of the gimbal while in motion.The user may select a gimbal mode before the vehicle is in motion. Theuser may be able to select a gimbal mode while the vehicle is in motion(e.g., alter the gimbal mode from an earlier gimbal mode). The user maythus affect the type of aerial photography that may be performed whilethe vehicle and/or gimbal are in the active mode.

The user may be able to select a gimbal operation mode while the gimbaland/or vehicle are in a simulation mode. In the simulation mode, theoperation mode of the gimbal may be selected by the user and permit theuser to undergo targeted training for a specific gimbal stabilizationmode. For instance, if the user wishes to practice gimbal control of agimbal that normally operates in following mode, the user may select afollowing mode for the simulation and practice the gimbal control forthe following mode.

The visual display may show virtual images captured by the virtualcamera. This may provide feedback for a user that is practicing thegimbal mode in the simulation. Depending on the gimbal mode selected,different visual effects may be provided for the virtual images capturedby the virtual camera. For instance, in the first person view mode, theimage may show a stabilized pitch to retain a stabilized horizon, butmay still react to the pitch and roll, much like the view from anindividual that may be moving as the vehicle. In the following mode, theimage may show a stabilized image about the pitch axis and a roll axiswithout stabilizing a yaw axis, which may give the visual effect of apan (e.g., if the viewer was turning the viewer's head side to side).When the gimbal is operated in the free gimbal mode, the image may bestabilized about a pitch axis, a yaw axis, and a roll axis which mayresult in the effect that the image may be moving, but in a floating,undisturbed manner.

The camera may be supported by a gimbal that may be carried by avehicle, such as a UAV. Any description of a UAV may apply to any typeof aerial vehicle. The UAV may have one or more propulsion units thatmay permit the UAV to move about in the air. The one or more propulsionunits may enable the UAV to move about one or more, two or more, threeor more, four or more, five or more, six or more degrees of freedom. Insome instances, the UAV may be able to rotate about one, two, three ormore axes of rotation. The axes of rotation may be orthogonal to oneanother. The axes of rotation may remain orthogonal to one anotherthroughout the course of the UAV's flight. The axes of rotation mayinclude a pitch axis, roll axis, and/or yaw axis. The UAV may be able tomove along one or more dimensions. For example, the UAV may be able tomove upwards due to the lift generated by one or more rotors. In someinstances, the UAV may be capable of moving along a Z axis (which may beup relative to the UAV orientation), an X axis, and/or a Y axis (whichmay be lateral). The UAV may be capable of moving along one, two, orthree axes that may be orthogonal to one another.

In view of the above attitude properties or characteristics, vehicleposition data, which may include simulated attitude of the vehicle, maybe characterized as pitch_aircraft, roll_aircraft and yaw_aircraft. Theterm “pitch_aircraft” is a variable descriptive of the orientation ofthe aircraft relative to the pitch axis. The term “roll_aircraft” is avariable descriptive of the orientation of the aircraft relative to theroll axis. The term “yaw_aircraft” is a variable descriptive of theorientation of the aircraft relative to the yaw axis. Therefore, theaircraft position data (which may include the simulated aircraftattitude data) may be represented asstatus_aircraft_sim=(pitch_aircraft, roll_aircraft, yaw_aircraft). Asdiscussed before, in some embodiments, values of these variables may beobtained from the UAV in the simulation mode. For instance, these valuesmay be supplied by a vehicle control system or a flight control systemfor a UAV. In some embodiments, values of these variables may beobtained from the remote control system or computer system where thevalues of these variables may have been previously stored for thesimulation purpose. The values for these variables may be generated withaid of a computer system, without using any aircraft data.

Similarly, the gimbal control data related to the attitude data of thegimbal may be characterized as pitch_gimbal_real, roll_gimbal_real, andyaw_gimbal_real. The term “pitch_gimbal_real” is a variable descriptiveof the orientation of the gimbal in the pitch axis relative to theaircraft. The term “roll_gimbal_real” is a variable descriptive of theorientation of the gimbal in the roll axis relative to the aircraft. Theterm “yaw_gimbal_real” is a variable descriptive of the orientation ofthe gimbal in the yaw axis relative to the aircraft.

As discussed before, the gimbal mode according to the embodiments of theinvention may include a first person view mode, a following mode and/ora free gimbal mode. In each mode, the gimbal may be stabilized aboutvarious axes to allow the gimbal to remain stable. This may permit theimage capture device to keep horizontal.

FIG. 3 is a flow chart schematically illustrating a method of obtaininga simulated attitude of a gimbal under a first person view (“FPV”) modein accordance with an embodiment of the invention.

In the FPV mode, the yaw attitude and roll attitude of a lens of theimage capture device should be maintained as being consistent with anaircraft head, such as the head of the UAV. Thus, when the UAV rotatesabout a yaw axis or a roll axis with respect to an environment, theimage capture device may correspondingly rotate with respect to theenvironment. The image capture device may rotate about the yaw axis orthe roll axis the same amount as the UAV. Therefore, the gimbal may onlyneed to compensate the pitch attitude to make the gimbal stable in thepitch axis. Thus, even if the UAV rotates about a pitch axis withrespect to an environment (real or virtual), the image capture devicedoes not correspondingly rotate with respect to the environment. If theimage capture device does rotate about the pitch axis, the rotationwould be in response to a user, instead of a stabilization effect. Thus,any rotation of the image capture device about the pitch axis relativeto the environment may be independent of rotation of the UAV about thepitch axis relative to the environment. The method of obtaining thesimulated attitude of the gimbal under the FPV mode may include thefollowing steps.

The method of obtaining the simulated attitude of the gimbal may includeobtaining vehicle position data 301. The gimbal control data may also beobtained 302. A selection of a FPV mode for a gimbal mode may beobtained 303. Simulated gimbal response data for the FPV mode may begenerated based on the vehicle position data, the gimbal control data,and the selection of the FPV mode 304. The simulated gimbal responsedata may be transmitted to a remote control system 305.

The vehicle position data may be obtained 301. The vehicle position datamay include the simulated attitude of the vehicle. As discussed before,the vehicle may be a UAV and the vehicle position data may be receivedfrom a flight control system on-board the UAV, or a vehicle controlsystem on-board the vehicle, or any other system. In some embodiments,if the method is performed at the UAV, then the vehicle position datamay be received from the remote control system or a separate externaldevice (e.g., computer). As discussed before, the vehicle position datamay be represented as variables pitch_aircraft, roll_aircraft andyaw_aircraft.

The gimbal control data may be obtained 302. The gimbal control data mayinclude a real or virtual attitude of the gimbal. The gimbal controldata may optionally be generated at a gimbal control system. The gimbalcontrol data may be generated at the gimbal in real-time or nearreal-time and then be transmitted to the remote control system, theflight control system, the vehicle control system, or any other systemdescribed herein, for further operations based on different embodiments.As discussed before, the gimbal control data may be represented aspitch_gimbal_real, roll_gimbal_real, and yaw_gimbal_real.

A gimbal mode may be obtained 303 through, for example, a userinstruction. The gimbal mode selection may include a selection of a FPVmode. A mode selecting signal may be provided from a remote controlsystem. A user may make a selection of the FPV mode at the remotecontrol system. Alternatively, one or more processors may select the FPVmode from other modes. A mode selecting signal may indicate which modethe user would like the gimbal to enter into. This user selection may bemade by a click on a display device, such as one shown in FIG. 1, or bya push made by the user on a mode selection button on the gimbal. Theselected mode may be represented as “gimbal_mode.”

Simulated gimbal response data may be generated 304. The simulatedgimbal response data may be generated using a data fusion algorithm. Thedata fusion algorithm may be launched and the vehicle position data, thegimbal control data and the gimbal mode selection may be entered intothe data fusion algorithm as inputs. The resulting simulated gimbalresponse data, as output of the data fusion algorithm, may include thesimulated attitude data of the gimbal and may be represented asyaw_gimbal, roll_gimbal, and pitch_gimbal. An exemplary data fusionalgorithm may be provided as follows:

1) “yaw_gimbal” may be “yaw_gimbal_real,” because the yaw attitude ofgimbal would be maintained as being consistent with the yaw attitude ofthe aircraft head in the FPV mode. In other words, the real yaw attitudeof the gimbal and the simulated yaw attitude of the aircraft may be thesame;

2) “roll_gimbal” may be “roll_gimbal_real,” because the roll attitude ofgimbal is maintained as being consistent with the roll attitude ofaircraft head in the FPV mode. In other words, the real roll attitude ofthe gimbal and the simulated roll attitude of the aircraft may be thesame; and

3) “pitch_gimbal” may be “pitch_gimbal_real+addment (pitch_aircraft,gimbal_mode).” The function “addment( )” herein may represent a variablecompensatory angle to compensate the pitch direction of the gimbal whenthe gimbal is in the FPV mode. For example, if the pitch aircraft of theUAV is +10° relative to the pitch axis when the FPV mode is selected,then the value of the addment function is −10° relative to the pitchaxis to offset the pitch orientation of the UAV. In this manner, thepitch direction of gimbal would be maintained stable when the UAVaccelerates in a back-forth direction.

Based on the above data fusion algorithm, the simulated gimbal responsedata may be obtained and transmitted to a remote control system 305. Insome instances, the simulated gimbal response data may be transmitted toa remote controller of the remote control system, which in turn maytransmit the simulated gimbal response data to a display device forvisual display. Alternatively, the simulated gimbal response data may bedirectly transmitted to a display device, or the display device and theremote controller may be the same device.

In some embodiments, if the data fusion algorithm herein is performed atthe flight control system on-board the UAV, then the simulated gimbalresponse data may be transmitted from the UAV to the display device(optionally via the remote controller). In another embodiment, if thedata fusion algorithm herein is performed at the remote control system,then the simulated gimbal response data may be directly transmitted fromthe remote controller to the display device (e.g., via a wiredconnection or a wireless connection). If the display device and theremote controller are integrated with each other, the simulated gimbalresponse data may be automatically displayed on the display of theremote controller. In other embodiments, the data fusion algorithm maybe performed at a gimbal control system (e.g., on-board a gimbal orUAV), and the simulated gimbal response data may be transmitted to theremote control system. The data fusion algorithm may occur on anyexternal device (e.g., server, cloud, other UAV), and may be transmittedto the remote control system.

The display device may comprise a screen that may depict the simulationdata in a 2D or 3D rendering. The display device may be a mobile phone(e.g. smart phone), tablet, desktop computer, laptop computer, virtualreality headset, or a television or projector in communication with acomputer device. In some cases, the display device may comprise a touchscreen, an LCD screen, or a plasma screen.

FIG. 4 is a flow chart schematically illustrating a method of obtaininga simulated attitude of a gimbal under a following mode in accordancewith an embodiment of the invention.

In the following mode, the yaw attitude of a lens of the image capturedevice should be maintained a constant angle with respect to an aircrafthead, such as the head of the UAV. Thus, when the UAV rotates about ayaw axis with respect to an environment, the image capture device maycorrespondingly rotate with respect to the environment with the constantangle relative to the aircraft head. The image capture device may rotateabout the pitch or the roll axis the different amount than the UAV.Therefore, the gimbal may only need to compensate the pitch attitude androll attitude to make the gimbal stable. Thus, even if the UAV rotatesabout a pitch axis and a roll axis with respect to an environment (realor virtual), the image capture device does not correspondingly rotatewith respect to the environment. If the image capture device does rotateabout the pitch axis and the roll axis, the rotation would be inresponse to a user, instead of a stabilization effect. Thus, anyrotation of the image capture device about the pitch axis and rollattitude relative to the environment may be independent of rotation ofthe UAV about the pitch axis and roll attitude relative to theenvironment. The method of obtaining the simulated attitude of thegimbal under the following mode may include the following steps.

The method of obtaining the simulated attitude of the gimbal under thefollowing mode may include obtaining vehicle position 401. The gimbalcontrol data may also be obtained 402. A selection of a following modefor a gimbal mode may be obtained 403. Simulated gimbal response datafor the following mode may be generated based on the vehicle positiondata, the gimbal control data, and the selection of the following mode404. The simulated gimbal response data may be transmitted to a remotecontrol system 405.

The vehicle position data may be obtained 401. The vehicle position datamay include the simulated attitude of the vehicle. As discussed before,the vehicle may be a UAV and the vehicle position data may be receivedfrom the flight control system on-board the UAV, or any other system. Insome embodiments, if the method is performed at the UAV, then thevehicle position data may be received from the remote control system ora separate external device (e.g., computer). As discussed before, thevehicle position data may be represented as variables pitch_aircraft,roll_aircraft and yaw_aircraft.

The gimbal control data may be obtained 402. The gimbal control data mayinclude a real or virtual attitude of the gimbal. The gimbal controldata may optionally be generated at a gimbal control system. The gimbalcontrol data may be generated at the gimbal in real-time or nearreal-time and then be transmitted to the remote control system, theflight control system, or any other system described herein, for furtheroperations based on different embodiments. As discussed before, thegimbal control data may be represented as pitch_gimbal_real,roll_gimbal_real, and yaw_gimbal_real.

A gimbal mode may be obtained 403 through, for example, a userinstruction. The gimbal mode selection may include a selection of afollowing mode. A mode selecting signal may be provided from a remotecontrol system. A user may make a selection of the following mode at theremote control system. Alternatively, one or more processors may selectthe following mode from other modes. The mode selecting signal mayindicate which mode the user would like the gimbal to enter into. Thisuser selection may be made by a click on a display device, such as oneshown in FIG. 1, or by a push made by the user on a mode selectionbutton on the gimbal. The selected mode may be represented as“gimbal_mode.”

Simulated gimbal response data may be generated 404. The simulatedgimbal response data may be generated using a data fusion algorithm. Thedata fusion algorithm may be launched and the vehicle position data, thegimbal control data and the gimbal mode selection may be entered intothe data fusion algorithm as inputs. The resulting simulated gimbalresponse data, as output of the data fusion algorithm, may include thesimulated attitude data of the gimbal and may be represented asyaw_gimbal, roll_gimbal, and pitch_gimbal. An exemplary data fusionalgorithm may be provided as follows:

1) “yaw_gimbal” may be “yaw_gimbal_real+lockAircraft (yaw_aircraft,gimbal_mode′yaw_aircraft_modeClicked),” because a constant angle betweenthe gimbal and aircraft head should be maintained, the functionlockAircraft( ) may be introduced. The value of the function“lockAircraft” represents an angle between the aircraft and the yaw axisof the gimbal at the timing of enabling the following mode of thegimbal. By this angle addition, the difference of the yaw attitude ofthe gimbal and the yaw attitude of the UAV may be maintained in thefollowing mode;

2) “roll_gimbal” may be “roll_gimbal_real+addment (roll_aircraft,gimbal_mode).” The function “addment( )” herein may represent a variablecompensatory angle to compensate the roll direction of the gimbal whenthe gimbal is in the following mode. For example, if the roll_aircraftof the UAV is +10° relative to the roll axis when the following mode isenabled, then the value of the addment function is −10° relative to theroll axis.” Based on this offset operation, the roll attitude of thegimbal should be maintained as being consistent with the roll attitudeof aircraft head in the following mode; and

3) “pitch_gimbal” may be “pitch_gimbal_real+addment (pitch_aircraft,gimbal_mode).” The function “addment( )” herein may represent a variablecompensatory angle to compensate the pitch direction of the gimbal whenthe gimbal is in the following mode. For example, if the pitch_aircraftof the UAV is +10° relative to the pitch axis when the following mode isenabled, then the value of the addment function is −10° relative to thepitch axis. In this manner, the angle shift due to the entrance of thefollowing mode could be compensated and therefore the gimbal could bemaintained stable in the pitch direction.

Based on the above data fusion algorithm, the simulated gimbal responsedata may be obtained and transmitted to a remote control system 405. Insome instances, the simulated gimbal response data may be transmitted toa remote controller of the remote control system, which in turn, maytransmit the simulated gimbal response data to a display device forvisual display. Alternatively, the simulated gimbal response data may bedirectly transmitted to a display device, or the display device and theremote controller may be the same device.

In some embodiments, if the data fusion algorithm herein is performed atthe flight control system on-board the UAV, then the simulated gimbalresponse data may be transmitted from the UAV to the display device(optionally via the remote controller). In another embodiment, if thedata fusion algorithm herein is performed at the remote control system,then the simulated gimbal response data may be directly transmitted fromthe remote controller to the display device (e.g., via a wiredconnection or a wireless connection). If the display device and theremote controller are integrated with each other, the simulated gimbalresponse data may be automatically displayed on the display of theremote controller. In other embodiments, the data fusion algorithm maybe performed at a gimbal control system (e.g., on-board a gimbal orUAV), and the simulated gimbal response data may be transmitted to theremote control system. The data fusion algorithm may occur on anyexternal device (e.g., server, cloud, other UAV), and may be transmittedto the remote control system.

The display device may comprise a screen that may depict the simulationdata in a 2D or 3D rendering. The display device may be a mobile phone(e.g. smart phone), tablet, desktop computer, laptop computer, virtualreality headset, or a television or projector in communication with acomputer device. In some cases, the display device may comprise a touchscreen, an LCD screen, or a plasma screen.

FIG. 5 is a flow chart schematically illustrating a method of obtaininga simulated attitudes of a gimbal under a free gimbal mode in accordancewith an embodiment of the invention.

In the free gimbal mode, the yaw attitude of a lens of the image capturedevice should be locked. For example, if the lens points towardssoutheast when the free gimbal mode is set, then the gimbal should keeppointing toward southeast regardless of the movement of the aircraft,except that an intersection angle between the gimbal and the aircraftexceeds the limit of the gimbal. Thus, when the UAV rotates about a yawaxis, a roll axis, or a pitch axis with respect to an environment, theimage capture device may not correspondingly rotate with respect to theenvironment. The image capture device may rotate about the yaw axis, theroll axis, or the pitch axis the different amount than the UAV.Therefore, it is required that the gimbal compensate the yaw, roll, andpitch attitudes to make itself stable. Thus, even if the UAV rotatesabout a pitch axis, a yaw axis, or a roll axis with respect to anenvironment (real or virtual), the image capture device does notcorrespondingly rotate with respect to the environment. If the imagecapture device does rotate about the pitch axis, the roll axis, or theyaw axis, the rotation would be in response to a user, instead of astabilization effect. Thus, any rotation of the image capture deviceabout the pitch axis, the roll axis, or the yaw axis relative to theenvironment may be independent of rotation of the UAV about the pitchaxis, the roll axis, or the yaw axis relative to the environment. Themethod of obtaining the simulated attitude of the gimbal under the freegimbal may include the following steps.

The method of obtaining the simulated attitude of the gimbal may includeobtaining vehicle position data 501. The gimbal control data may also beobtained 502. A selection of a free gimbal for a gimbal mode may beobtained 503. Simulated gimbal response data for the free gimbal modemay be generated based on the vehicle position data, the gimbal controldata, and the selection of the free gimbal mode 504. The simulatedgimbal response data may be transmitted to a remote control system 505.

The vehicle position data may be obtained 501. The vehicle position datamay include the simulated attitude of the vehicle. As discussed before,the vehicle may be a UAV and the position data may be received from theflight control system on-board the UAV, or any other system. In someembodiments, if the method is performed at the UAV, then the vehicleposition data may be received from the remote control system or aseparate external device (e.g., computer). As discussed before, theposition data may be represented as variables pitch_aircraft,roll_aircraft and yaw_aircraft.

The gimbal control data may be obtained 502. The gimbal control data mayinclude a real or virtual attitude of the gimbal. The gimbal controldata may optionally be generated at a gimbal control system. The gimbalcontrol data may be generated at the gimbal in real-time or nearreal-time and then be transmitted to the remote control system, theflight control system, or any other system described herein, for furtheroperations based on different embodiments. As discussed before, thegimbal control data may be represented as pitch_gimbal_real,roll_gimbal_real, and yaw_gimbal_real.

A gimbal mode may be obtained 503 through, for example, a userinstruction. The gimbal mode selection may include a selection of afollowing gimbal mode. A mode selecting signal may be provided from aremote control system. A user may make a selection of the free gimbalmode at the remote control system. Alternatively, one or more processorsmay select the free gimbal mode from other modes. The mode selectingsignal may indicate which mode the user would like the gimbal to enterinto. This user selection may be made by a click on a display device,such as one shown in FIG. 1, or by a push made by the user on a modeselection button on the gimbal. The selected mode may be represented as“gimbal_mode.”

Simulated gimbal response data may be generated 504. The simulatedgimbal response data may be generated using a data fusion algorithm. Thedata fusion algorithm may be launched and the vehicle position data, thegimbal control data and the gimbal mode selection may be entered intothe data fusion algorithm as inputs. The resulting simulated gimbalresponse data, as output of the data fusion algorithm, may include thesimulated attitude data of the gimbal and may be represented asyaw_gimbal, roll_gimbal, and pitch_gimbal. An exemplary data fusionalgorithm may be provided as follows:

1) “yaw_gimbal” may be “yaw_gimbal_real+Addment (yaw_aircraft,gimbal_mode, yaw_aircraft_modeClicked),” because the yaw_gimbal shouldbe kept constant in the free gimbal mode, the function Addment( ) may beintroduced. The value of the function “Addment” represents an anglebetween the aircraft and the yaw axis of the gimbal at the timing ofenabling the free gimbal mode of the gimbal. By this angle addition, thedifference of the yaw attitude of the gimbal and the yaw attitude of theUAV may be maintained unchanged in the free gimbal mode;

2) “roll_gimbal” may be “roll_gimbal_real+addment (roll_aircraft,gimbal_mode).” The function “addment( )” herein may represent a variablecompensatory angle to compensate the roll direction of the gimbal whenthe gimbal is in the free gimbal mode. For example, if the roll_aircraftof the UAV is +10° relative to the roll axis when the free gimbal modeis enabled, then the value of the addment function is −10° relative tothe roll axis. In this way, the roll attitude of the gimbal should bemaintained unchanged in the free gimbal mode; and

3) “pitch_gimbal” may be “pitch_gimbal_real+addment (pitch_aircraft,gimbal_mode).” The function “addment( )” herein may represent a variablecompensatory angle to compensate the pitch direction of the gimbal whenthe gimbal is in the free gimbal mode. For example, if thepitch_aircraft of the UAV is +10° relative to the pitch axis when thefree gimbal mode is enabled, then the value of the addment function is−10° relative to the pitch axis. In this manner, the angle shift due tothe entrance of the free gimbal mode could be compensated and thereforethe gimbal could be maintained stable in the pitch direction.

Based on the above data fusion algorithm, the simulated gimbal responsedata may be obtained and transmitted to a remote control system 505. Insome instances, the simulated gimbal response data may be transmitted toa remote controller of the remote control system, which in turn, maytransmit the simulated gimbal response data to a display device forvisual display. Alternatively, the simulated gimbal response data may bedirectly transmitted to a display device, or the display device and theremote controller may be the same device.

In some embodiments, if the data fusion algorithm herein is performed atthe flight control system on-board the UAV, then the simulated gimbalresponse data may be transmitted from the UAV to the display device(optionally via the remote controller). In another embodiment, if thedata fusion algorithm herein is performed at the remote control system,then the simulated gimbal response data may be directly transmitted fromthe remote controller to the display device (e.g., via a wiredconnection or a wireless connection). If the display device and theremote controller are integrated with each other, the simulated gimbalresponse data may be automatically displayed on the display of theremote controller. In other embodiments, the data fusion algorithm maybe performed at a gimbal control system (e.g., on-board a gimbal orUAV), and the simulated gimbal response data may be transmitted to theremote control system. The data fusion algorithm may occur on anyexternal device (e.g., server, cloud, other UAV), and may be transmittedto the remote control system.

The display device may comprise a screen that may depict the simulationdata in a 2D or 3D rendering. The display device may be a mobile phone(e.g. smart phone), tablet, desktop computer, laptop computer, virtualreality headset, or a television or projector in communication with acomputer device. In some cases, the display device may comprise a touchscreen, an LCD screen, or a plasma screen.

Any of the modes described herein may be used to effect simulation ofgimbal control for a user. In other instances, the various modes may beutilized during an active mode, such as when a user is actually flying aUAV and/or controlling a gimbal to collect image data using the imagecapture device. The user may select from different available modes toobtain different types of aerial photography.

FIG. 6 is a flow chart schematically illustrating a method of obtaininga simulated attitude of a gimbal when environment factors are taken intoaccount in accordance with an embodiment of the invention.

In some embodiments, the vehicle position data describing an attitude ofthe vehicle from a vehicle control system on-board the vehicle is atleast partially based on simulated weather data. In an example, theweather data is directed to data regarding one or more of a wind speed,a wind direction, precipitation (e.g., rain, snow, sleet, hail),humidity, air density, and/or temperature. In some embodiments, a usermay select a weather condition through a user interface provided on thedisplay device. A user may select a weather condition from a pluralityof available weather conditions. Alternatively or in addition, theweather condition may be provided as an input to a physical model forapplying a constant or variable force on the gimbal, for example a heador tail wind. For instance, the user may input a direction of windand/or speed of wind. The weather conditions may optionally be selectedor generated at random without requiring user input. The weatherconditions may be generated based on previously collected real-lifeweather information. The weather conditions may be generated based oncurrent weather conditions determined by one or more environmentalsensors. Alternatively, the weather conditions may be entirely virtuallygenerated without requiring any real-world data.

The physical model may output physical simulation data. The physicalsimulation data may be one or more physical parameters that may act onthe gimbal. In an example, the physical model may output one or moreforces acting on the gimbal. The simulated physical forces may have asimulated effect on the virtual gimbal. This may affect positioning ofthe virtual gimbal (e.g., orientation of the gimbal about a yaw, pitch,and/or roll axis, spatial location of the gimbal), which may affect thesimulated gimbal response data.

As discussed before with respect to FIGS. 3-5, the gimbal may be in oneof multiple modes, such as the FPV mode, the following mode and the freegimbal mode and therefore different data fusion algorithms are appliedfor these modes. However, these algorithms are not taken intoenvironment factors into account. Thus, as an alternative or additionalembodiment, FIG. 6 illustrates a general data fusion algorithm forobtaining the simulated gimbal response data when taking the environmentfactors into account.

The functions and operations 601, 602, and 603 are similar to thosediscussed with respect to 301, 302 and 303 in FIGS. 3, 401, 402, and 403in FIG. 4, and 501, 502, and 503 in FIG. 5. The simulated weatherinformation may be used in combination with any of the modes describedherein.

The method of obtaining a simulated attitude of a gimbal when takingenvironment factors into account may include obtaining the vehicleposition data 601. The gimbal control data may also be obtained 602. Agimbal mode may be obtained 603. The weather conditions may be obtained604. Simulated gimbal response data for the FPV mode may be generatedbased on the vehicle position data, the gimbal control data, and theselected mode and weather conditions 605. The simulated gimbal responsedata may be transmitted to a remote control system 606.

The vehicle position data may be obtained 601. The vehicle position datamay include the simulated attitude of the vehicle. As discussed before,the vehicle may be a UAV and the position data may be received from aflight control system on-board the UAV, or any other system. In someembodiments, if the method is performed at the UAV, then the vehicleposition data may be received from the remote control system or aseparate external device (e.g., computer). As discussed before, thevehicle position data may be represented as variables pitch_aircraft,roll_aircraft and yaw_aircraft.

The gimbal control data may be obtained 602. The gimbal control data mayinclude a real or virtual attitude of the gimbal. The gimbal controldata may optionally be generated at a gimbal control system. The gimbalcontrol data may be generated at the gimbal in real-time or nearreal-time and then be transmitted to the control system, the flightcontrol system, or any other system described herein, for furtheroperations based on different embodiments. As discussed before, thegimbal control data may be represented as pitch_gimbal_real,roll_gimbal_real, and yaw_gimbal_real.

A gimbal mode may be obtained 603 through e.g., a user instruction. Auser may make a selection of the gimbal mode at the remote controlsystem. Alternatively, one or more processors may select the gimbal modefrom other modes. A gimbal mode selecting signal may indicate which modethe user would like the gimbal to enter into. This user selection may bemade by a click on a display device, such as one shown in FIG. 1, or bya push made by the user on a mode selection button on the gimbal. Theselected mode may be represented as “gimbal_mode.”

The environment factors or conditions may be obtained 604. For example,the environment factors may be a wind speed, a wind direction,precipitation (e.g., type of precipitation, amount of precipitation),humidity, air density, amount of sunshine, cloudiness, and/ortemperature. In some instances, each of these factors may be determinedindividually and/or independently. In some instances, one or more ofthese factors may be bundled together to form one or more preselectedweather modes. For instance, a user may select a “winter” weather mode,where there may be snow, moderate winds, low temperature, low sunshine.In another example, a user may select a “windy” mode with high windspeeds which may come from changing or unpredictable directions, and mayhave sunshine and no precipitation. In another example, a user mayselect “night flight” mode where the skies may be clear, the temperaturelow, and no sunshine. Thus, various flight condition options may bepresented to a user that a user may select, where each flight conditionoption may have the factors preselected. These factors may be chosenrandomly by a physical model or they may be input or selected by a user.In some cases, variable properties of the environment may be derivedfrom a prior real flight performed by the UAV or other UAVs, or currentreal data from one or more sensors.

A user may select a weather condition through the user interfaceprovided on the display device. The weather condition may be provided asan input to generate simulated gimbal response data 605. In an example,a weather condition may result in a constant or variable force on thegimbal. For example a head or tail wind may provide forces fromdifferent directions. The forces may be any one or combination of a dragforce, lift force, gravitational force, normal force, tangential force,or any other force known to act on the gimbal. The wind speed may affectthe amount of force that the gimbal may experience. For instance, higherwind speeds may exert a greater degree of force on the gimbal, and thesimulated gimbal response data may reflect this. In another example,precipitation may also affect the gimbal. For instance, a downpour mayprovide a downward force on the gimbal. The response of the gimbal toinstructions may be different under different weather conditions. Forexample, excessive heat or cold may affect how a gimbal may respond to acontrol signal. For instance, the gimbal may respond more slowly or notto as great a degree.

Then, according to the different gimbal mode, one or more of theyaw_gimbal_real, roll_gimbal_real, and pitch_gimbal_real may becompensated by taking the weather conditions into account 605.

For example, if the gimbal is in the free gimbal mode, then“roll_gimbal” may be “roll_gimbal_real+addment (roll_aircraft,gimbal_mode)+roll_gimbal_real_weather.” The function “addment( )” hereinmay represent a variable compensatory angle to compensate the rolldirection of the gimbal when the gimbal is in the following mode and thenew entry roll_gimbal_real_weather denotes an angle difference in theroll axis caused by the current weather. For example, if theroll_aircraft of the UAV is +20° relative to the roll axis when thefollowing mode is enabled and the weather is set to be a regular wind,then the value of the addment function is −20° relative to the rollaxis. In this way, the roll attitude of the gimbal should be maintainedas being consistent with the roll attitude of aircraft head in thefollowing mode.

Further, “pitch_gimbal” may be “pitch_gimbal_real+addment(pitch_aircraft, gimbal_mode)+pitch_gimbal_real_weather” in the freegimbal mode. The function “addment( )” herein may represent a variablecompensatory angle to compensate the pitch direction of the gimbal whenthe gimbal is in the following mode and the new entrypitch_gimbal_real_weather denotes an angle difference in the pitch axiscaused by the current weather. For example, if the pitch_aircraft of theUAV is +20° relative to the pitch axis when the following mode isenabled and the current weather is set to be a baffling wind, then thevalue of the addment function is −20° relative to the pitch axis. Inthis manner, the angle shift due to the entrance of the following modeand the current weather could be compensated and therefore the gimbalcould be maintained stable in the pitch direction.

Based on the above data fusion algorithm, the simulated gimbal responsedata taking the weather conditions into account may be obtained andtransmitted to a remote control system 606. In some instances, thesimulated gimbal response data may be transmitted to a remote controllerof the remote control system, which in turn, may transmit the simulatedgimbal response data to a display device for visual display.Alternatively, the simulated gimbal response data may be directlytransmitted to a display device, or the display device and the remotecontroller may be the same device.

In some embodiments, if the data fusion algorithm herein is performed atthe flight control system on-board the UAV, then the simulated gimbalresponse data may be transmitted from the UAV to the display device(optionally, via the remote controller). In another embodiment, if thedata fusion algorithm herein is performed at the controller system, thenthe simulated gimbal response data may be directly transmitted from theremote controller to the display device (e.g., via a wired connection ora wireless connection). If the display device and the remote controllerare integrated with each other, the simulated gimbal response data maybe automatically displayed on the display of the remote controller. Inother embodiments, the data fusion algorithm may be performed at agimbal control system (e.g., on-board a gimbal or UAV), and thesimulated gimbal response data may be transmitted to the remote controlsystem. The data fusion algorithm may occur on any external device(e.g., server, cloud, other UAV), and may be transmitted to the remotecontrol system.

The display device may comprise a screen that may depict the simulationdata in a 2D or 3D rendering. The display device may be a mobile phone(e.g. smart phone), tablet, desktop computer, laptop computer, virtualreality headset, or a television or projector in communication with acomputer device. In some cases, the display device may comprise a touchscreen, an LCD screen, or a plasma screen.

It is to be understood that the mode selection and the weather conditionselection in accordance with some embodiments of the invention asdiscussed before with respect to FIGS. 3-6 are only for illustrativepurposes. A person skilled in the art may understand, based on theteaching of the specification, that the simulation methods, systems anddevices of the invention may be practiced without the mode selection andthe weather condition selection. For example, the simulation method asproposed may be implemented merely based on a single mode, such as oneof the FPV mode, the following mode and the free gimbal mode, which maybe preset or selected by the user. Therefore, the user may be trainedunder a specific mode, which may make the training more targeted.Likewise, the weather condition selection may be ignored or omitted ifthe real flying environment is ideal and peaceful. Further, the datafusion directed to different gimbal modes herein is merely illustrativeof some examples of the simulation algorithms and the simulation methodsof the invention may be applied without regard to the gimbal mode.

FIG. 7 is a flow chart schematically illustrating a method of simulatinggimbal control in accordance with an embodiment of the invention.

Simulated gimbal response data generated by a gimbal control systemon-board the vehicle may be received 701 at a remote control systemremote to a vehicle, wherein the simulated gimbal response data isgenerated based on (1) gimbal control data from the remote controlsystem configured to communicate with the gimbal control system and (2)position data describing an attitude of the vehicle generated from avehicle control system on-board the vehicle. A simulated gimbalrepresentation may be displayed at the remote control system based onthe simulated gimbal response data on a display device 702.

In some embodiments, the vehicle is an unmanned aerial vehicle (“UAV”)and the simulated gimbal response data is determined according to agimbal mode, wherein the gimbal mode includes one of a first person viewmode, a following mode, or a free gimbal mode as discussed in detailbefore with respect to FIGS. 3-5.

When the gimbal is operated in the first person view mode, the simulatedgimbal response data stabilizes a pitch axis of the gimbal with respectto the environment without stabilizing a yaw axis and a roll axis. Inother words, the pitch attitude of the gimbal would be compensated bythe data fusion taking into account the environment, such as the weatherdata, as discussed with respect to FIG. 6.

When the gimbal is operated in the following mode, the simulated gimbalresponse data stabilizes a pitch axis and a roll axis of the gimbal withrespect to the environment without stabilizing a yaw axis. In otherwords, the pitch and roll attitudes of the gimbal would be compensatedby the data fusion taking into account the environment, such as theweather data, as discussed with respect to FIG. 6.

When the gimbal is operated in the free gimbal mode, the simulatedgimbal response data stabilizes a pitch axis, a yaw axis, and a rollaxis of the gimbal with respect to the environment. In other words, thepitch and roll attitudes of the gimbal would be compensated by the datafusion taking into account the environment, such as the weather data, asdiscussed with respect to FIG. 6.

The gimbal response data may be determined based on a gimbal mode signalgenerated at the remote control system remote to the vehicle. The gimbalmode signal may be generated in response to a user input indicating aselection of a gimbal mode from the plurality of gimbal modes, such asthe first person view mode, the following mode and the free gimbal mode.A user may select a gimbal mode from a plurality of gimbal modes. Thesimulation may reflect the selected gimbal mode. Virtual visual datacaptured by the virtual camera may reflect the selected gimbal mode. Auser may alter a gimbal mode via a user input. The alteration to thegimbal mode may occur prior to the simulated flight and/or during thesimulated flight. The simulation may reflect the update to the gimbalmode. The virtual visual data captured by the virtual camera may beupdated to reflect the updated gimbal mode.

A remote controller may be configured to transmit the simulated gimbalresponse data to a display device comprising a visual display.Alternatively, the display device may directly obtain the simulatedgimbal response data. The visual display may show simulated gimbal stateinformation of the UAV. The simulated gimbal state information mayinclude simulated visual data captured by a camera fitted in the gimbal.The display device may be a mobile device and the remote control systemmay communicate with the display device via a wireless connection. Insome embodiments, the remote control system may comprise the displaydevice. In some instances, a remote controller that may accept a userinput to control the gimbal and/or UAV may have a display deviceincorporated therein. In an example, the remote control system may bethe same remote control system used to operate the vehicle and thegimbal in a real flight operation.

The remote control system may further comprise a display device thatreceives the simulated gimbal response data and displays a visualillustration of the gimbal in an orientation described by the gimbalresponse data. The remote control system may include one or morejoystick controls useful for controlling directional heading of thegimbal.

The position data describing an attitude of the vehicle from a vehiclecontrol system on-board the vehicle may be at least partially based onsimulated orientation of the vehicle. In some embodiments, the positiondata describing an attitude of the vehicle from a vehicle control systemon-board the vehicle is at least partially based on simulated weatherdata.

In some embodiments, the position data describing an attitude of thevehicle from a vehicle control system on-board the vehicle is at leastpartially based on the position data describing the simulated attitudeof the vehicle which includes at least one of (1) rotation of thevehicle about a pitch axis, (2) rotation of the vehicle about a yawaxis, or (3) rotation of the vehicle about a roll axis.

FIG. 8 is a flow chart schematically illustrating a method of simulatinggimbal control in accordance with an embodiment of the invention.

As illustrated in FIG. 8, a gimbal mode signal indicative of a selectionfrom a plurality of gimbal modes is obtained 801. Then, gimbal controldata from a remote control system and vehicle position data describing asimulated attitude of the vehicle generated from a vehicle controlsystem on-board the vehicle are obtained 802. Simulated gimbal responsedata is generated 803 based on the gimbal control data, the vehicleposition data describing the simulated attitude of the vehicle, and thegimbal mode signal, wherein the simulated gimbal response data causes adifferent set of axes to be stabilized with respect to an environment ofthe vehicle under each of the plurality of gimbal modes.

As discussed before with respect to FIGS. 2-6, the gimbal mode accordingto the embodiments of the invention may include a first person viewmode, a following mode and/or a free gimbal mode. In each mode, thegimbal may be stabilized about various axes to allow the gimbal toremain stable. This may permit the image capture device to keephorizontal.

In some embodiments, the simulated gimbal response data may includesimulated gimbal state information which represents an attitude of thegimbal relative to the vehicle. A visual display may be used to showsimulated gimbal state information of the vehicle. Further, thesimulated gimbal state information may include simulated visual datacaptured by a camera fitted in the gimbal.

The vehicle position data describing an attitude of the vehicle from avehicle control system on-board the vehicle may be at least partiallybased on simulated orientation of the vehicle. In some embodiments, theposition data describing an attitude of the vehicle from a vehiclecontrol system on-board the vehicle is at least partially based onsimulated weather data. The vehicle position data describing an attitudeof the vehicle from a vehicle control system on-board the vehicle may beat least partially based on the vehicle position data describing thesimulated attitude of the vehicle which includes at least one of (1)rotation of the vehicle about a pitch axis, (2) rotation of the vehicleabout a yaw axis, or (3) rotation of the vehicle about a roll axis.

As discussed before, based on the gimbal control data, the vehicleposition data, and the gimbal mode signal, and optional weather data, adata fusion algorithm may be performed at a gimbal control system (e.g.,on-board a gimbal or UAV. The data fusion algorithm may occur on anyexternal device (e.g., server, cloud, other UAV), and may be transmittedto the remote control system. In some embodiments, the method mayfurther comprise transmitting the simulated gimbal response data to aremote controller of the remote control system, which in turn maytransmit the simulated gimbal response data to a display device thatdisplays a visual illustration of the gimbal in an orientation describedby the gimbal response data. Alternatively, the simulated gimbalresponse data may be directly transmitted to a display device, or thedisplay device and the remote controller may be the same device. Thesimulated gimbal response data may be generated by one or more meprocessors.

FIG. 9 is a schematic diagram of a gimbal on-board a vehicle inaccordance with an embodiment of the invention.

As illustrated in FIG. 9, the gimbal 900 on-board the vehicle mayinclude a receiver 901 configured to receive a gimbal mode signalindicative of a selection from a plurality of gimbal modes. The gimbal900 on-board the vehicle may further comprise a gimbal control system902 configured to (1) receive gimbal control data from a remote controlsystem, (2) receive position data describing a simulated attitude of thevehicle generated from a flight control system on-board the vehicle and(3) simulated gimbal response data based on (1) the gimbal control data,(2) the position data describing the simulated attitude of the vehicle,and (3) the gimbal mode signal, wherein the simulated gimbal responsedata causes a different set of axes to be stabilized with respect to anenvironment of the vehicle under each of the plurality of gimbal modes.

The gimbal may optionally comprise one or more gimbal components thatmay be movable relative to one another. The gimbal components may moverelative to one another with aid of an actuator. Each gimbal componentmay optionally have a corresponding actuator that may permit themovement of the gimbal component. In some instances, one or more gimbalcomponents may permit rotation of the payload about a pitch axis, one ormore gimbal components may permit rotation of the payload about a yawaxis, and one or more gimbal components may permit rotation of thepayload about a roll axis. In some instances, depending on a selectedmode, the gimbal components may be individually controlled to provide adesired stabilization effect. For instance, a first gimbal component maybe stabilized while a second and a third gimbal component are notstabilized. In another instance, a first and second gimbal componentsmay be stabilized, while the third gimbal component is not stabilized.In another example, the first, second, and third gimbal components mayall be stabilized. The payload may be supported by one or more of thegimbal components. The payload may directly contact a single gimbalcomponent. Alternatively, the payload may directly contact multiplegimbal components.

When the gimbal mode signal is indicative of a selection of the firstperson view mode, the simulated gimbal response data may stabilize apitch axis of the gimbal with respect to the environment withoutstabilizing a yaw axis and a roll axis. In other words, the pitchattitude of the gimbal would be compensated by the data fusion takinginto account the environment, such as the weather data, as discussedwith respect to FIG. 6.

When the gimbal mode signal is indicative of a selection of thefollowing mode, the simulated gimbal response data may stabilize a pitchaxis and a roll axis of the gimbal with respect to the environmentwithout stabilizing a yaw axis. In other words, the pitch and rollattitudes of the gimbal would be compensated by the data fusion takinginto account the environment, such as the weather data, as discussedwith respect to FIG. 6.

When the gimbal mode signal is indicative of a selection of the freegimbal mode, the simulated gimbal response data stabilizes a pitch axis,a yaw axis, and a roll axis of the gimbal with respect to theenvironment. In other words, the pitch and roll attitudes of the gimbalwould be compensated by the data fusion taking into account theenvironment, such as the weather data, as discussed with respect to FIG.6.

In some embodiments, the gimbal mode signal may be generated at theremote control system remote to the vehicle. In some embodiments, thegimbal mode signal may be generated in response to a user inputindicating a selection of a gimbal mode from the plurality of gimbalmodes

In some embodiments, (1) gimbal control data from a remote controlsystem configured to communicate with the gimbal control system and (2)position data describing a simulated attitude of the vehicle arereceived at a gimbal control system on-board a vehicle. The vehicle maybe an unmanned aerial vehicle (“UAV”). The simulated gimbal stateinformation may include simulated visual data captured by a camerafitted in the gimbal.

The simulated attitude of the vehicle may be generated by a vehiclecontrol system on-board the vehicle and a remote control system isconfigured to receive the simulated gimbal response data and transmitthe simulated gimbal response data to a display device comprising avisual display showing simulated gimbal state information of the UAV.The remote control system may be the same remote control system used tooperate the vehicle and the gimbal in a real flight operation and mayinclude one or more joystick controls useful for controlling directionalheading of the gimbal.

In some embodiments, the display device may be a mobile device and theremote control system may communicate with the display device via awireless connection. In some embodiments, the remote control system maycomprise the display device that receives the simulated gimbal responsedata and displays a visual illustration of the gimbal in an orientationdescribed by the gimbal response data.

FIG. 10 is a flow chart schematically illustrating a method of operatinga gimbal on-board a vehicle in accordance with an embodiment of theinvention.

As illustrated in FIG. 10, a gimbal mode signal indicative of whetherthe gimbal is to be in an active mode or a simulation mode is obtained1001. Gimbal control data from a remote control system is obtained 1002.Gimbal response data is generated 1003 at the gimbal control systembased on the gimbal control data from the remote control system, whereinthe gimbal response data is (1) communicated to one or more actuatorsconfigured to adjust an arrangement of the gimbal when the gimbal is inthe active mode and is (2) not communicated to one or more actuatorswhen the gimbal is in the simulation mode.

The simulated gimbal response data may include simulated gimbal statedata which represents an attitude of the gimbal relative to the vehicle.A visual display may show simulated gimbal state information of thevehicle. The simulated gimbal state information may include simulatedvisual data captured by a camera fitted in the gimbal.

In some embodiments, the method may further comprise transmitting thesimulated gimbal response data to a display device that displays avisual illustration of the gimbal in an orientation described by thegimbal response data. The simulated gimbal response data may begenerated by one or more processors.

The position data describing an attitude of the vehicle from a vehiclecontrol system on-board the vehicle may be at least partially based onsimulated orientation of the vehicle. Further, the position datadescribing an attitude of the vehicle from the vehicle control systemon-board the vehicle may be at least partially based on simulatedweather data. The position data describing an attitude of the vehiclefrom the vehicle control system on-board the vehicle may at leastpartially based on the position data describing the simulated attitudeof the vehicle which includes at least one of (1) rotation of thevehicle about a pitch axis, (2) rotation of the vehicle about a yawaxis, or (3) rotation of the vehicle about a roll axis.

In some embodiments, the one or more actuators include one or moremotors. The one or more actuators may permit rotation of one or moreframe components of the gimbal about an axis. The one or more actuatorsmay permit translation of the gimbal. In some embodiments, the gimbalmode signal may be generated at the remote control system remote to thevehicle or may be generated in response to a user input indicating aselection between the active mode and the simulation mode.

FIG. 11 is a schematic diagram of a gimbal on-board a vehicle inaccordance with an embodiment of the invention.

As illustrated in FIG. 11, the gimbal 1100 on-board the vehicle mayinclude a receiver 1101, configured to receive a gimbal mode signalindicative of whether the gimbal is to be in an active mode or asimulation mode. The gimbal 1100 on-board the vehicle may furtherinclude a gimbal control system 1102 configured to (1) receive gimbalcontrol data from a remote control system, and (2) gimbal response databased on the gimbal control data from the remote control system. Thegimbal 1100 on-board the vehicle may additionally include one or moreactuators 1103 configured to (1) adjust an arrangement of the gimbalwhen the gimbal is in the active mode, or (2) remain dormant and notadjust the arrangement of the gimbal when the gimbal is in thesimulation mode.

The gimbal control data from a remote control system configured tocommunicate with the gimbal control system and (2) position datadescribing a simulated attitude of the vehicle are received at a gimbalcontrol system on-board the vehicle are received at a control systemon-board the vehicle. The remote control system may be configured toreceive the simulated gimbal response data and transmit the simulatedgimbal response data to a display device comprising a visual displayshowing simulated gimbal state information of the vehicle. The remotecontrol system may include one or more joystick controls useful forcontrolling directional heading of the gimbal. The display device may bea mobile device and the remote control system may communicate with thedisplay device via a wireless connection.

The remote control system may comprise a display device that receivesthe simulated gimbal response data and displays a visual illustration ofthe gimbal in an orientation described by the gimbal response data. Theremote control system may be the same remote control system used tooperate the vehicle and the gimbal in a real flight operation. In someembodiments, the simulated gimbal state information includes simulatedvisual data captured by a camera fitted in the gimbal.

In some instances, the position data describing an attitude of thevehicle from a vehicle control system on-board the vehicle may be atleast partially based on simulated orientation of the vehicle. Further,the position data describing an attitude of the vehicle from a vehiclecontrol system on-board the vehicle may be at least partially based onsimulated weather data.

The position data describing an attitude of the vehicle from a vehiclecontrol system on-board the vehicle may be at least partially based onthe position data describing the simulated attitude of the vehicle whichincludes at least one of (1) rotation of the vehicle about a pitch axis,(2) rotation of the vehicle about a yaw axis, or (3) rotation of thevehicle about a roll axis.

In some embodiments, the one or more actuators include one or moremotors. The one or more actuators may permit rotation of one or moreframe components of the gimbal about an axis. The one or more actuatorsmay permit translation of the gimbal. In some embodiments, the gimbalmode signal may be generated at the remote control system remote to thevehicle. The gimbal mode signal may be generated in response to a userinput indicating a selection between the active mode and the simulationmode. The vehicle herein may be an unmanned aerial vehicle (“UAV”).

FIG. 12 is a flow chart schematically illustrating a method ofsimulating gimbal control in accordance with an embodiment of theinvention.

As illustrated in FIG. 12, (1) gimbal control data from a remote controlsystem configured to communicate with the gimbal control system and (2)vehicle position data describing a simulated attitude of the vehicle areobtained 1201. Simulated gimbal response data is generated 1202 at agimbal control system based on the gimbal control data and the positiondata describing the simulated attitude of the vehicle. Then, thesimulated gimbal response data and the simulated attitude of the vehicleare transmitted to a remote control system 1203, which may in turn,transmit the simulated attitude of the vehicle to a display device,wherein the display device may generate a visual depiction based on thesimulated gimbal response data and the simulated attitude of thevehicle. The vehicle here may be an unmanned aerial vehicle (“UAV”).

In some instances, the simulated gimbal response data may includesimulated gimbal state data which represents an attitude of the gimbalrelative to the vehicle. A visual display may show simulated gimbalstate information of the vehicle. The simulated gimbal state informationincludes simulated visual data captured by a camera fitted in thegimbal. The remote control system may include one or more joystickcontrols useful for controlling directional heading of the gimbal. Thesimulated gimbal response data may be generated by one or moreprocessors.

In some embodiments, the position data describing an attitude of thevehicle from a vehicle control system on-board the vehicle may be atleast partially based on simulated orientation of the vehicle. In someembodiments, the position data describing an attitude of the vehiclefrom a vehicle control system on-board the vehicle may be at leastpartially based on simulated weather data descriptive of multipleweather conditions. The position data describing an attitude of thevehicle from a vehicle control system on-board the vehicle may be atleast partially based on the position data describing the simulatedattitude of the vehicle which includes at least one of (1) rotation ofthe vehicle about a pitch axis, (2) rotation of the vehicle about a yawaxis, or (3) rotation of the vehicle about a roll axis.

FIG. 13 is a schematic diagram of a gimbal simulation system inaccordance with an embodiment of the invention.

As illustrated in FIG. 13, the gimbal simulation system 1300 may includea gimbal 1301 on-board a vehicle. The gimbal simulation system 1300 mayfurther comprise a gimbal control system 1302 on-board the vehicleconfigured to (1) receive gimbal control data from a remote controlsystem, (2) receive position data describing a simulated attitude of thevehicle generated from a vehicle control system on-board the vehicle;and (3) generate simulated gimbal response data based on (i) the gimbalcontrol data and (ii) the position data describing the simulatedattitude of the vehicle. The gimbal simulation system 1300 mayadditionally include a communication unit 1303 configured to transmit,to a display device, the simulated gimbal response data and thesimulated attitude of the vehicle, wherein the display device generatesa visual depiction based on the simulated gimbal response data and thesimulated attitude of the vehicle.

The display device is a handheld device. The display device may beintegrated with a remote control system in communication with the gimbalcontrol system. In some embodiments, the simulated gimbal response datais calculated by determining a simulated attitude of the gimbal relativeto the simulated attitude of the vehicle.

In some embodiments, (1) the gimbal control data from a remote controlsystem configured to communicate with the gimbal control system and (2)the position data describing a simulated attitude of the vehicle arereceived at a gimbal control system on-board a vehicle. The vehicle maybe an unmanned aerial vehicle (“UAV”). The simulated attitude of thevehicle may be generated by a vehicle control system on-board thevehicle.

In some embodiments, the simulated gimbal response data may bedetermined according to a gimbal mode from multiple gimbal modes,wherein the gimbal mode may include one of a first person view mode, afollowing mode, and a free gimbal mode. When the gimbal is operated inthe first person view mode, the simulated gimbal response data maystabilize a pitch axis of the gimbal with respect to the environmentwithout stabilizing a yaw axis and a roll axis. When the gimbal isoperated in the following mode, the simulated gimbal response data maystabilize a pitch axis and a roll axis of the gimbal with respect to theenvironment without stabilizing a yaw axis. When the gimbal is operatedin the free gimbal mode, the simulated gimbal response data maystabilize a pitch axis, a yaw axis, and a roll axis of the gimbal withrespect to the environment. The environment herein may be in respect tomultiple weather conditions.

In some instances, the position data describing an attitude of thevehicle from a vehicle control system on-board the vehicle may be atleast partially based on simulated orientation of the vehicle. In someembodiments, the position data describing an attitude of the vehiclefrom a vehicle control system on-board the vehicle may be at leastpartially based on simulated weather data. The position data describingan attitude of the vehicle from a vehicle control system on-board thevehicle may be at least partially based on the position data describingthe simulated attitude of the vehicle which includes at least one of (1)rotation of the vehicle about a pitch axis, (2) rotation of the vehicleabout a yaw axis, or (3) rotation of the vehicle about a roll axis.

A simulation may provide a user with a virtual experience of controllinga gimbal while a UAV is in simulated flight. The user may practicecontrolling the gimbal alone, or may practice controlling the gimbal andthe UAV. In some instances, different users may practice controllingeach part (e.g., a first user may control a gimbal while a second usermay control a UAV).

The simulation may show the user the perspective of an image capturedevice supported by the gimbal. The user may view virtual visual datacaptured by the image capture device in real-time. Depending on aselected gimbal mode, the virtual image capture device may be stabilizedabout different rotational axes which may provide a different visualeffect by the image capture device.

The simulation may also show the gimbal operating in the virtualenvironment. The simulation may show the gimbal supported by the UAVwithin the virtual environment. The location of the gimbal and/or UAV inthe virtual environment may be displayed. The positioning of the gimbalmay be displayed. For instance, the degree of rotation by one or moregimbal components may be displayed. The orientation of the payloadwithin the virtual environment may be displayed. In some instances,virtual measurements may be displayed. For instance, degree oforientation of the UAV about one or more axes may be displayed asnumerical values. Similarly, a spatial location of the UAV in thevirtual environment may be displayed as spatial coordinates. In asimilar manner, the degree of orientation of the payload supported bythe gimbal about one or more axes may be displayed as numerical values.The spatial location of the payload in the virtual environment may bedisplayed as spatial coordinates. Measurements for degree of rotationfor each gimbal component may or may not be displayed as numericalvalues.

The virtual environment may be displayed. The virtual environment mayinclude one or more geographic features. This may include naturallyoccurring features such as mountains, hills, lakes, rivers, creeks,valleys, forests, boulders, terrain, shores, or any other naturallyoccurring feature. This may also include virtual manmade features suchas buildings, vehicles, bridges, airports, landing strips, or any otherman-made feature. Moving objects such as humans or animals may bedisplayed. The virtual image from the image capturing device maycorrespond to the estimated position and orientation of the imagecapture device within the virtual environment.

FIG. 14 schematically illustrates an unmanned aerial vehicle 1400 with agimbal in accordance with an embodiment of the invention.

The UAV may be an example of a movable object as described herein. TheUAV 1400 may include a propulsion system having four rotors, such asthose 1402, 1404, and 1406 as explicitly depicted. Any number of rotorsmay be provided (e.g., one, two, three, four, five, six, or more). Therotors, rotor assemblies, or other propulsion systems of the unmannedaerial vehicle may enable the unmanned aerial vehicle to hover/maintainposition, change orientation, and/or change location. The distancebetween shafts of opposite rotors may be any suitable length. Forexample, the length may be less than or equal to 2 m, or less than equalto 5 m. In some embodiments, the length may be within a range from 40 cmto 1 m, from 10 cm to 2 m, or from 5 cm to 5 m. Any description hereinof a UAV may apply to a movable object, such as a movable object of adifferent type, and vice versa. The UAV may use an assisted takeoffsystem or method as described herein.

In some embodiments, the movable object may be configured to carry aload, such as the gimbal 1408, which may support an image capture device1410 as depicted. The load may include one or more of passengers, cargo,equipment, instruments, and the like. The load may be provided within ahousing. The housing may be separate from a housing of the movableobject, or be part of a housing for a movable object. Alternatively, theload may be provided with a housing while the movable object does nothave a housing. Alternatively, portions of the load or the entire loadmay be provided without a housing. The load may be rigidly fixedrelative to the movable object. Optionally, the load may be movablerelative to the movable object (e.g., translatable or rotatable relativeto the movable object). The load may include a payload and/or a carrier,as described elsewhere herein.

In some embodiments, the movement of the movable object, carrier, andpayload relative to a fixed reference frame (e.g., the surroundingenvironment) and/or to each other, may be controlled by a terminal. Theterminal may be a remote controller device, such as the remotecontroller 101 shown in FIG. 1, at a location distant from the movableobject, carrier, and/or payload. The terminal may be disposed on oraffixed to a support platform. Alternatively, the terminal may be ahandheld or wearable device. For example, the terminal may include asmart phone, tablet, laptop, computer, glasses, gloves, helmet,microphone, or suitable combinations thereof. The terminal may include auser interface, such as a keyboard, mouse, joystick, touch screen, ordisplay. Any suitable user input may be used to interact with theterminal, such as manually entered commands, voice control, gesturecontrol, or position control (e.g., via a movement, location or tilt ofthe terminal).

The terminal may be used to control any suitable state of the movableobject, carrier, and/or payload. For example, the terminal may be usedto control the position and/or orientation of the movable object,carrier, and/or payload relative to a fixed reference from and/or toeach other. In some embodiments, the terminal may be used to controlindividual elements of the movable object, carrier, and/or payload, suchas the actuation assembly of the carrier, a sensor of the payload, or anemitter of the payload. The terminal may include a wirelesscommunication device adapted to communicate with one or more of themovable object, carrier, or payload. The terminal may correspond to aremote controller as described elsewhere herein. The terminal mayoptionally be a display device or may include a display device. Theterminal may be part of the remote control system.

The terminal may include a suitable display unit for viewing informationof the movable object, carrier, and/or payload. For example, theterminal may be configured to display information of the movable object,carrier, and/or payload with respect to position, translationalvelocity, translational acceleration, orientation, angular velocity,angular acceleration, or any suitable combinations thereof. In someembodiments, the terminal may display information provided by thepayload, such as data provided by a functional payload (e.g., imagesrecorded by a camera or other image capturing device).

Optionally, the same terminal may both control the movable object,carrier, and/or payload, or a state of the movable object, carrierand/or payload, as well as receive and/or display information from themovable object, carrier and/or payload. For example, a terminal maycontrol the positioning of the payload relative to an environment, whiledisplaying image data captured by the payload, or information about theposition of the payload. Alternatively, different terminals may be usedfor different functions. For example, a first terminal may controlmovement or a state of the movable object, carrier, and/or payload whilea second terminal may receive and/or display information from themovable object, carrier, and/or payload. For example, a first terminalmay be used to control the positioning of the payload relative to anenvironment while a second terminal displays image data captured by thepayload. Various communication modes may be utilized between a movableobject and an integrated terminal that both controls the movable objectand receives data, or between the movable object and multiple terminalsthat both control the movable object and receives data. For example, atleast two different communication modes may be formed between themovable object and the terminal that both controls the movable objectand receives data from the movable object.

FIG. 15 illustrates a movable object 1500 including a carrier 1502 and apayload 1504, in accordance with embodiments of the invention. Althoughthe movable object 1500 is depicted as an aircraft, this depiction isnot intended to be limiting, and any suitable type of movable object maybe used, as previously described herein. One of skill in the art wouldappreciate that any of the embodiments described herein in the contextof aircraft systems may be applied to any suitable movable object (e.g.,a UAV). In some instances, the payload 1504 may be provided on themovable object 1500 without requiring the carrier 1502. The movableobject 1500 may include propulsion mechanisms 1506, a sensing system1508, and a communication system 1510.

The propulsion mechanisms 1506 may include one or more of rotors,propellers, blades, engines, motors, wheels, axles, magnets, or nozzles,as previously described. The movable object may have one or more, two ormore, three or more, or four or more propulsion mechanisms. Thepropulsion mechanisms may all be of the same type. Alternatively, one ormore propulsion mechanisms may be different types of propulsionmechanisms. The propulsion mechanisms 1506 may be mounted on the movableobject 1500 using any suitable means, such as a support element (e.g., adrive shaft) as described elsewhere herein. The propulsion mechanisms1506 may be mounted on any suitable portion of the movable object 1500,such on the top, bottom, front, back, sides, or suitable combinationsthereof.

In some embodiments, the propulsion mechanisms 1506 may enable themovable object 1500 to take off vertically from a surface or landvertically on a surface without requiring any horizontal movement of themovable object 1500 (e.g., without traveling down a runway). Optionally,the propulsion mechanisms 1506 may be operable to permit the movableobject 1500 to hover in the air at a specified position and/ororientation. One or more of the propulsion mechanisms 1506 may becontrolled independently of the other propulsion mechanisms.Alternatively, the propulsion mechanisms 1506 may be configured to becontrolled simultaneously. For example, the movable object 1500 may havemultiple horizontally oriented rotors that may provide lift and/orthrust to the movable object. The multiple horizontally oriented rotorsmay be actuated to provide vertical takeoff, vertical landing, andhovering capabilities to the movable object 1500. In some embodiments,one or more of the horizontally oriented rotors may spin in a clockwisedirection, while one or more of the horizontally oriented rotors mayspin in a counterclockwise direction. For example, the number ofclockwise rotors may be equal to the number of counterclockwise rotors.The rotation rate of each of the horizontally oriented rotors may bevaried independently in order to control the lift and/or thrust producedby each rotor, and thereby adjust the spatial disposition, velocity,and/or acceleration of the movable object 1500 (e.g., with respect to upto three degrees of translation and up to three degrees of rotation).

The sensing system 1508 may include one or more sensors that may sensethe spatial disposition, velocity, and/or acceleration of the movableobject 1500 (e.g., with respect to up to three degrees of translationand up to three degrees of rotation). The one or more sensors mayinclude global positioning system (“GPS”) sensors, motion sensors,inertial sensors, proximity sensors, or image sensors. The sensing dataprovided by the sensing system 1508 may be used to control the spatialdisposition, velocity, and/or orientation of the movable object 1500(e.g., using a suitable processing unit and/or control module, asdescribed below). Alternatively, the sensing system 1508 may be used toprovide data regarding the environment surrounding the movable object,such as weather conditions, proximity to potential obstacles, locationof geographical features, location of manmade structures, and the like.

The communication system 1510 enables communication with terminal 1512having a communication system 1514 via wireless signals 1516. Thecommunication systems 1510, 1514 may include any number of transmitters,receivers, and/or transceivers suitable for wireless communication. Thecommunication may be one-way communication, such that data may betransmitted in only one direction. For example, one-way communicationmay involve only the movable object 1500 transmitting data to theterminal 1512, or vice-versa. The data may be transmitted from one ormore transmitters of the communication system 1510 to one or morereceivers of the communication system 1512, or vice-versa.Alternatively, the communication may be two-way communication, such thatdata may be transmitted in both directions between the movable object1500 and the terminal 1512. The two-way communication may involvetransmitting data from one or more transmitters of the communicationsystem 1510 to one or more receivers of the communication system 1514,and vice-versa.

In some embodiments, the terminal 1512 may provide control data to oneor more of the movable object 1500, carrier 1502, and payload 1504 andreceive information from one or more of the movable object 1500, carrier1502, and payload 1504 (e.g., position and/or motion information of themovable object, carrier or payload; data sensed by the payload such asimage data captured by a payload camera). In some instances, controldata from the terminal may include instructions for relative positions,movements, actuations, or controls of the movable object, carrier and/orpayload. For example, the control data may result in a modification ofthe location and/or orientation of the movable object (e.g., via controlof the propulsion mechanisms 1506), or a movement of the payload withrespect to the movable object (e.g., via control of the carrier 1502).The control data from the terminal may result in control of the payload,such as control of the operation of a camera or other image capturingdevice (e.g., taking still or moving pictures, zooming in or out,turning on or off, switching imaging modes, change image resolution,changing focus, changing depth of field, changing exposure time,changing viewing angle or field of view).

In some instances, the communications from the movable object, carrierand/or payload may include information from one or more sensors (e.g.,of the sensing system 1508 or of the payload 1504). The communicationsmay include sensed information from one or more different types ofsensors (e.g., GPS sensors, motion sensors, inertial sensor, proximitysensors, or image sensors). Such information may pertain to the position(e.g., location, orientation), movement, or acceleration of the movableobject, carrier and/or payload. Such information from a payload mayinclude data captured by the payload or a sensed state of the payload.The control data provided and transmitted by the terminal 1512 may beconfigured to control a state of one or more of the movable object 1500,carrier 1502, or payload 1504. Alternatively or in combination, thecarrier 1502 and payload 1504 may also each include a communicationmodule configured to communicate with terminal 1512, such that theterminal may communicate with and control each of the movable object1500, carrier 1502, and payload 1504 independently.

In some embodiments, the movable object 1500 may be configured tocommunicate with another remote device in addition to the terminal 1512,or instead of the terminal 1512. The terminal 1512 may also beconfigured to communicate with another remote device as well as themovable object 1500. For example, the movable object 1500 and/orterminal 1512 may communicate with another movable object, or a carrieror payload of another movable object. When desired, the remote devicemay be a second terminal or other computing device (e.g., computer,laptop, tablet, smartphone, or other mobile device). The remote devicemay be configured to transmit data to the movable object 1500, receivedata from the movable object 1500, transmit data to the terminal 1512,and/or receive data from the terminal 1512. Optionally, the remotedevice may be connected to the Internet or other telecommunicationsnetwork, such that data received from the movable object 1500 and/orterminal 1512 may be uploaded to a website or server.

In some embodiments, in a simulation mode, the movable object 1500 maybe physically separated from the carry 1502 and the payload 1504. Thecarry 1502 and the payload 1504 may be controlled by the terminal 1512to perform the gimbal control simulation as discussed throughout theinvention. In some embodiments, in the simulation mode, the movableobject 1500 may be set to participate in the gimbal control simulation.As discussed before with respect to the drawings, a vehicle controlsystem and a gamble control system may be operated within the moveableobject 1500 to perform data fusion of the gamble control data and theposition data of the moveable object such that the simulated attitudedata of the gimbal could be obtained.

FIG. 16 is a schematic illustration by way of block diagram of a system1600 for controlling a movable object or carrier, in accordance withembodiments. The system 1600 may be used in combination with anysuitable embodiment of the systems, devices, and methods disclosedherein. The system 1600 may include a sensing module 1602, a processingunit 1604, non-transitory computer readable medium 1606, a controlmodule 1608, and a communication module 1610.

The sensing module 1602 may utilize different types of sensors thatcollect information relating to the movable objects in different ways.Different types of sensors may sense different types of signals orsignals from different sources. For example, the sensors may includeinertial sensors, GPS sensors, proximity sensors (e.g., lidar), orvision/image sensors (e.g., a camera). The sensing module 1602 may beoperatively coupled to a processing unit 1604 having a plurality ofprocessors. In some embodiments, the sensing module 1602 may beoperatively coupled to a transmission module 1612 (e.g., a Wi-Fi imagetransmission module) configured to directly transmit sensing data to asuitable external device or system. For example, the transmission module1612 may be used to transmit images captured by a camera of the sensingmodule 1602 to a remote terminal.

The processing unit 1604 may have one or more processors, such as aprogrammable processor (e.g., a central processing unit (CPU)). Theprocessing unit 1604 may be operatively coupled to a non-transitorycomputer readable medium 1606. The non-transitory computer readablemedium 1606 may store logic, code, and/or program instructionsexecutable by the processing unit 1604 for performing one or more stepsas discussed before. The non-transitory computer readable medium mayinclude one or more memory units (e.g., removable media or externalstorage such as an SD card or random access memory (RAM)). In someembodiments, data from the sensing module 1602 may be directly conveyedto and stored within the memory units of the non-transitory computerreadable medium 1606. The memory units of the non-transitory computerreadable medium 1606 may store logic, code and/or program instructionsexecutable by the processing unit 1604 to perform any suitableembodiment of the methods described herein. For example, the processingunit 1604 may be configured to execute instructions causing one or moreprocessors of the processing unit 1604 to analyze sensing data producedby the sensing module 1602. The memory units may store sensing data fromthe sensing module 1602 to be processed by the processing unit 1604. Insome embodiments, the memory units of the non-transitory computerreadable medium 1606 may be used to store the processing resultsproduced by the processing unit 1604.

In some embodiments, the processing unit 1604 may be operatively coupledto a control module 1608 configured to control a state of the movableobject. For example, the control module 1608 may be configured tocontrol the propulsion mechanisms of the movable object to adjust thespatial disposition, velocity, and/or acceleration of the movable objectwith respect to six degrees of freedom. Alternatively or in combination,the control module 1608 may control one or more of a state of a carrier,payload, or sensing module.

The processing unit 1604 may be operatively coupled to a communicationmodule 1610 configured to transmit and/or receive data from one or moreexternal devices (e.g., a terminal, display device, or other remotecontroller). Any suitable means of communication may be used, such aswired communication or wireless communication. For example, thecommunication module 1610 may utilize one or more of local area networks(“LANs”), wide area networks (“WANs”), infrared, radio, WiFi,point-to-point (“P2P”) networks, telecommunication networks, cloudcommunication, and the like. Optionally, relay stations, such as towers,satellites, or mobile stations, may be used. Wireless communications maybe proximity dependent or proximity independent. In some embodiments,line-of-sight may or may not be required for communications. Thecommunication module 1610 may transmit and/or receive one or more ofsensing data from the sensing module 1602, processing results producedby the processing unit 1604, predetermined control data, user commandsfrom a terminal or remote controller, and the like.

The components of the system 1600 may be arranged in any suitableconfiguration. For example, one or more of the components of the system1600 may be located on the movable object, carrier, payload, terminal,sensing system, or an additional external device in communication withone or more of the above. Additionally, although FIG. 16 depicts asingle processing unit 1604 and a single non-transitory computerreadable medium 1606, one of skill in the art would appreciate that thisis not intended to be limiting, and that the system 1600 may include aplurality of processing units and/or non-transitory computer readablemedia. In some embodiments, one or more of the plurality of processingunits and/or non-transitory computer readable media may be situated atdifferent locations, such as on the movable object, carrier, payload,terminal, sensing module, additional external device in communicationwith one or more of the above, or suitable combinations thereof, suchthat any suitable aspect of the processing and/or memory functionsperformed by the system 1600 may occur at one or more of theaforementioned locations.

While preferred embodiments of the invention have been shown anddescribed herein, it will be obvious to those skilled in the art thatsuch embodiments are provided by way of example only. Numerousvariations, changes, and substitutions will now occur to those skilledin the art without departing from the invention. It should be understoodthat various alternatives to the embodiments of the invention describedherein may be employed in practicing the invention. It is intended thatthe following claims define the scope of the invention and that methodsand structures within the scope of these claims and their equivalents becovered thereby.

What is claimed is:
 1. A gimbal simulation system comprising: a gimbal control system on-board a vehicle configured to (1) receive gimbal control data from a remote control system, (2) receive position data describing a simulated attitude of the vehicle generated from a vehicle control system on-board the vehicle; and (3) generate simulated gimbal response data based on (i) the gimbal control data and (ii) the position data describing the simulated attitude of the vehicle; and a communication unit configured to transmit the simulated gimbal response data to the remote control system.
 2. The gimbal simulation system of claim 1 wherein the vehicle is an unmanned aerial vehicle (UAV).
 3. The gimbal simulation system of claim 1 wherein the simulated gimbal response data is determined according to a gimbal mode, wherein the gimbal mode includes one of a first person view mode, a following mode, or a free gimbal mode.
 4. The gimbal simulation system of claim 3 wherein the simulated gimbal response data stabilizes a pitch axis of the gimbal with respect to an environment without stabilizing a yaw axis and a roll axis, when the gimbal is operated in the first person view mode.
 5. The gimbal simulation system of claim 3 wherein the simulated gimbal response data stabilizes a pitch axis and a roll axis of the gimbal with respect to an environment without stabilizing a yaw axis, when the gimbal is operated in the following mode.
 6. The gimbal simulation system of claim 3 wherein the simulated gimbal response data stabilizes a pitch axis, a yaw axis, and a roll axis of the gimbal with respect to an environment, when the gimbal is operated in the free gimbal mode.
 7. The gimbal simulation system of claim 1 wherein the simulated gimbal response data is determined based on a gimbal mode signal generated at the remote control system remote to the vehicle.
 8. The gimbal simulation system of claim 7 wherein the gimbal mode signal is generated in response to a user input indicating a selection of a gimbal mode from the plurality of gimbal modes.
 9. The gimbal simulation system of claim 1 wherein the remote control system is configured to transmit the simulated gimbal response data to a display device comprising a visual display.
 10. The gimbal simulation system of claim 9 wherein the remote control system communicates with the display device via a wireless connection.
 11. The gimbal simulation system of claim 9 wherein the remote control system is used to operate the vehicle and the gimbal on-board the vehicle in a real flight operation.
 12. The gimbal simulation system of claim 9 wherein the visual display shows simulated gimbal state information of the vehicle.
 13. The gimbal simulation system of claim 12 wherein the simulated gimbal state information includes simulated visual data captured by a camera coupled to the gimbal.
 14. The gimbal simulation system of claim 1 wherein the system further comprises a display device that receives the simulated gimbal response data and displays a visual illustration of the gimbal in an orientation described by the simulated gimbal response data.
 15. The gimbal simulation system of claim 1 wherein the remote control system is configured to control directional heading of the gimbal.
 16. The gimbal simulation system of claim 1 wherein the position data is at least partially based on simulated orientation of the vehicle.
 17. The gimbal simulation system of claim 1 wherein the position data is at least partially based on simulated weather data.
 18. The gimbal simulation system of claim 1 wherein the position data is at least partially based on position data describing the simulated attitude of the vehicle which includes at least one of (1) rotation of the vehicle about a pitch axis, (2) rotation of the vehicle about a yaw axis, or (3) rotation of the vehicle about a roll axis.
 19. A method of simulating gimbal control, said method comprising: receiving, (1) gimbal control data from a remote control system configured to communicate with a gimbal control system on-board a vehicle and (2) position data describing a simulated attitude of the vehicle; generating, at the gimbal control system, simulated gimbal response data based on the gimbal control data and the position data describing the simulated attitude of the vehicle; and transmitting, to the remote control system, the simulated gimbal response data from the gimbal control system on-board the vehicle.
 20. A gimbal on-board a vehicle, comprising: a receiver, configured to receive a gimbal mode signal indicative of whether the gimbal is to be in an active mode or a simulation mode; a gimbal control system configured to (1) receive gimbal control data from a remote control system, and (2) generate gimbal response data based on the gimbal control data from the remote control system; and one or more actuators configured to (1) adjust an arrangement of the gimbal when the gimbal is in the active mode, or (2) remain dormant and not adjust the arrangement of the gimbal when the gimbal is in the simulation mode.
 21. The gimbal of claim 20 wherein the gimbal control system is further configured to (1) receive position data describing a simulated attitude of the vehicle generated from a vehicle control system on-board the vehicle in the simulation mode and (2) generate simulated gimbal response data based on the gimbal control data and the position data.
 22. The gimbal of claim 21 wherein the simulated gimbal response data includes simulated gimbal state information which represents an attitude of the gimbal relative to the vehicle.
 23. The gimbal of claim 22 wherein the simulated gimbal state information includes simulated visual data captured by a camera fitted in the gimbal.
 24. The gimbal of claim 21 wherein the simulated gimbal response data is transmitted to a display device that displays a visual illustration of the gimbal in an orientation described by the simulated gimbal response data.
 25. The gimbal of claim 21 wherein the position data is at least partially based on simulated orientation of the vehicle or simulated weather data.
 26. The gimbal of claim 21 wherein the position data is at least partially based on position data describing the simulated attitude of the vehicle which includes at least one of (1) rotation of the vehicle about a pitch axis, (2) rotation of the vehicle about a yaw axis, or (3) rotation of the vehicle about a roll axis.
 27. The gimbal of claim 20 wherein the one or more actuators permit rotation of one or more frame components of the gimbal about an axis in the active mode.
 28. The gimbal of claim 20 wherein the gimbal mode signal is generated at the remote control system remote to the vehicle or generated in response to a user input indicating a selection between the active mode and the simulation mode.
 29. The gimbal of claim 20 wherein the vehicle is an unmanned aerial vehicle (UAV).
 30. A method of operating a gimbal on-board a vehicle, said method comprising: receiving a gimbal mode signal indicative of whether the gimbal is to be in an active mode or a simulation mode; receiving gimbal control data from a remote control system; and generating, at the gimbal control system, gimbal response data based on the gimbal control data from the remote control system, wherein the gimbal response data is (1) communicated to one or more actuators configured to adjust an arrangement of the gimbal when the gimbal is in the active mode and is (2) not communicated to one or more actuators when the gimbal is in the simulation mode. 